Advice for Debugging Apache with PHP Segmentation Faults

Question

Every hour or so an Apache child process seg. faults on our web server. We are running a non-threaded PHP 5.2.17 Apache module with Apache prefork MPM. I have run httpd with a few core dumps, gdb and this .gdbinit file from php's github repo, typing this command into gdb for each:

dump_bt executor_globals.current_execute_data 

With no experience with gdb or the inner-workings of the php interpreter, I can't make anything from the results.

core.22762 [0x53896ef0] () :-2118682552 [0x538977a0] () /Statement/Interface.php:113 [0x538978a0] /Zend/Db/Statement/Interface.php() core.22791 [0x538977a0] () @:0 [0x538978a0] () core.5568 [0x53896ef0] () :2061035360 [0x538977a0] () :1767992432 [0x538978a0] () core.30384 [0x538977a0] () :0 [0x538978a0] () core.3091 [0x53896ef0] mysql_query():992424253 [0x538977a0] () ~:17 [0x538978a0] () 

(core.3091 also shows this after Program terminated with signal 11, Segmentation fault.)

#0 0x00002b6e7ad8d67d in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff53896ef0) at /usr/src/debug/php-5.2.17/Zend/zend_vm_execute.h:217 217 EX_T(opline->result.u.var).var.fcall_returned_reference = return_reference; 

Is there something else that I can type in gdb to get a better stack trace? Is there anything else that you do when debugging Apache segmentation faults?

Any help would be greatly appreciated, thanks.

Answer 1

I've seen this before.. apart from getting your PHP app developers to check their code, and making sure you have up-to-date versions (from your distribution) of PHP and apache, there isn't much you can do.

If you're running an Op-Code cacher (eaccelerator, APC or XCache), you could try turning them off, they're known to cause the odd segfault.

In the past, we've had to deploy some impressively hackish solutions to a segfaulting system, for short-term fixes, of course. For example, this cronjob worked well:

# Restart apache when a segfault is found in the most recent line of errorlog #*/5 * * * * tail -n1 /var/log/apache2/error.log | grep 'Segmentation fault' && /etc/init.d/apache2 restart

Or you could try something a bit more clever: #*/5 * * * * /usr/bin/wget http://www.my-site.com/ -T10 -O 2> /dev/null - | grep "Hosted by" > /dev/null || /usr/local/bin/brutally_restart_apache.sh

Where brutally_restart is:

 #!/bin/sh /usr/sbin/apache2ctl stop sleep 6 killall -9 apache2 sleep 4 /usr/sbin/apache2ctl start