Today we had interesting issue.
One server completely stalled, due to file system crash. I mean – reboot, shutdown -r now, init 6 not working.
In this case your can use magic keys (SysRq) remotely via /proc/sys/trigger

  echo b > /proc/sysrq-trigger - send Alt + SysRq + B to kernel

Alt + SysRq + B – Makes Immediate reboot without sync of file systems, action is the same as RESET pressed.

Additional information:
http://en.wikipedia.org/wiki/Magic_SysRq_key
Thanks to out colleague Yuriy for this information. Useful.

How to get youngest revision version

  ~$svnlook youngest /var/lib/svnmirror

How to check repo (Damn it takes huge ammount of time) more then 3 month for 301914 revisions. And CPU usege runs high, so if your store content at SVN (it`s really not a good idea…) an your need to veryfy it, better to do verifycation on dedicated powerful server.

  
  ~$svnadmin verify /home/svn/content/

Add all new created files to SVN

 ~$svn st | grep "^?" | awk '{print $2}' | xargs svn add

As usual…

After some changes apache start producing segfauls.

#vi /etc/apache2/apache2.conf

CoreDumpDirectory /tmp/apache2-cores

#ulimit -c unlimited

#/etc/init.d/apache2 restart

Now apache will put core files to /tmp/apache2-cores (don`t forget to create folder with sufficient rights.)

Testing

#kill -11 PID
#ls /tmp/apache2-cores

All woring, we can observer core.

Now launch your ugly script causing cores.

Run GDB.

#gdb /usr/sbin/apache2 /tmp/apache2-cores/core

Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
[New process 26683]
#0 0xf7662caf in memcpy () from /lib/libc.so.6
(gdb) where
#0 0xf7662caf in memcpy () from /lib/libc.so.6
#1 0xf63ef13f in restore_class () from /usr/lib/php5/20060613+lfs/eaccelerator.so
#2 0xf63ed390 in eaccelerator_compile_file () from /usr/lib/php5/lfs/eaccelerator.so
#3 0xf6987e73 in compile_filename () from /usr/lib/apache2/modules/libphp5.so
#4 0xf69d2bbf in ?? () from /usr/lib/apache2/modules/libphp5.so
#5 0x00000008 in ?? ()
#6 0xfff5de20 in ?? ()
#7 0xfff5dda8 in ?? ()
#8 0xf698aa8e in ?? () from /usr/lib/apache2/modules/libphp5.so
#9 0x08bfe23d in ?? ()
#10 0x08bfd240 in ?? ()
#11 0xfff5dda4 in ?? ()
#12 0xf69c912e in ?? () from /usr/lib/apache2/modules/libphp5.so
#13 0xf52edc24 in ?? ()
#14 0xfff5e024 in ?? ()
#15 0x00000000 in ?? ()
(gdb)

This mean segfault occurs at eaccelerator. When eaccelerator tries to put bytecode to disk.
Damn. As usual all accelerators are crap. 🙁

Disable eaccelerator at /etc/php5/conf-d/xxxx.ini
and restart apache.
Error gone. According to my experience and experience of my colleges, all accelerators creates segfault on some applications. If your want stability, better to extend number of servers. But if accelerator is working for you – your are lucky guy! 🙂

Some time ago, I start my IT wiki ksimute.trancom.ru, but it`s in russian language.

So only few people can read it. 🙁 I decide to continue writing in english, to share knowlage with more people. And to move data from my former job domain to my own domain.

I really hope that this blog will help somebody 🙂 in their job.  Because it`s usually helps me to solve issues.

P.S. Excuse my english. It`s realy bad, but usually everyone understand, what I`am talking about.

Sergey.