PHP-FPM and continuos "exited on signal 11"

Question

I've a Hp web server with Xeon E5504 2ghz and 8gb ram, with freebsd 8.2-p9 amd64, nginx 1.2.1, PHP 5.3.14, mysql 5.5.25, apc, memcached and other package installed using freebsd ports. (conf file at the end of that message)

My trouble are simple, I've a lot of "exited on signal 11" in /var/log/messages and when php-fpm child die I've "502 gateway error" on my website, that's really boring because I've a lot of that errors in a day, in that php-fpm conf I've forced it to an heavy configuration with a 150 of php-fpm's child in static mode, and pm.max_requests = 10000000, but in my php-fpm log I found for each child:

[19-Jul-2012 18:58:14.666913] NOTICE: pid 84563, fpm_children_make(), line 421: [pool www] child 84717 started [19-Jul-2012 18:58:14.666984] DEBUG: pid 84563, fpm_event_loop(), line 409: event module triggered 1 events [19-Jul-2012 18:58:17.403217] DEBUG: pid 84563, fpm_event_loop(), line 409: event module triggered 2 events [19-Jul-2012 18:58:17.407442] DEBUG: pid 84563, fpm_got_signal(), line 72: received SIGCHLD [19-Jul-2012 18:58:17.407552] WARNING: pid 84563, fpm_children_bury(), line 252: [pool www] child 84563 exited on signal 11 (SIGSEGV) after 39.849444 seconds from start 

and in /var/log/message for the same pid:

Jul 19 18:58:14 web1 kernel: pid 84563 (php-fpm), uid 1001: exited on signal 11 

to be sure that isn't a configuration files (php.ini and php-fpm.conf) troubles, I restored it with the original one that I found in the archive, but nothing, than I tried to use spawn-fcgi, but same trouble!

Than I tried to generate a php-fpm.core running "gdb /usr/local/sbin/php-fpm", when the child exit with "exited on signal 11" php-fpm main process exit with "exited on signal 5 (core dumped)" (that only when I run it throw GDB!), here logs and gdb backtrace results:

• /var/log/messages (see above)
• php-fpm.log
• gdb

As you can see in last link, that are last rows:

#659 0x0000000801534de0 in ?? () from /lib/libc.so.7 #660 0x0000000000000001 in ?? () #661 0x00007fffffffec08 in ?? () #662 0x000000000000000f in ?? () Cannot access memory at address 0x800000000000 

Any ideas?

Thanks!

/var/log/messages:

Jul 24 17:58:41 web1 kernel: pid 3887 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:42 web1 kernel: pid 3998 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:42 web1 kernel: pid 3895 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:42 web1 kernel: pid 3892 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:43 web1 kernel: pid 3889 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:43 web1 kernel: pid 3898 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:44 web1 kernel: pid 3886 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:44 web1 kernel: pid 3999 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:45 web1 kernel: pid 3896 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:45 web1 kernel: pid 4000 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:45 web1 kernel: pid 3893 (php-fpm), uid 1001: exited on signal 11 Jul 24 17:58:55 web1 kernel: pid 3885 (php-fpm), uid 0: exited on signal 5 (core dumped) 

php.ini (only difference from original file)

[PHP] memory_limit = 128M apc.enabled=1 apc.shm_size=128M apc.ttl=0 apc.mmap_file_mask=/tmp/apc/apc.XXXXXX error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE display_errors = On display_startup_errors = On track_errors = On html_errors = On upload_max_filesize = 4M [Date] date.timezone = Europe/Rome [browscap] browscap = /home/serverweb/etc/php/browscap.ini 

php-fpm.conf (only difference from original file)

[global] error_log = /home/serverweb/log/php-fpm/error.log log_level = debug emergency_restart_threshold = 10 emergency_restart_interval = 1m process_control_timeout = 10s process.max = 0 [www] user = web group = web listen = 127.0.0.1:9999 listen.allowed_clients = 127.0.0.1 pm = static pm.max_children = 16 pm.max_requests = 10000000 slowlog = /home/serverweb/log/php-fpm/$pool.log.slow 

Answer 1

PHP 5.3.14 has a serious bug that leads to segfault in php-fpm. It was fixed in 5.3.15 and 5.4.5.

Answer 2

If it's still happening in the recent version of PHP, try running:

$ dmesg | grep segfault | tail -10 php-fpm[327]: segfault at 834ac30 ip 000000000834ac30 sp 00007ffc112d1b78 error 14 in libnss_nis-2.23.so[7fe02b461000+b000] php-fpm[329]: segfault at 834ac30 ip 000000000834ac30 sp 00007ffc112d1b78 error 14 php-fpm[331]: segfault at 834ac30 ip 000000000834ac30 sp 00007ffc112d2278 error 14 in libnss_nis-2.23.so[7fe02b461000+b000] 

At the end of the lines you can see in which library the crash is happening, e.g.

error 14 in libnss_nis-2.23.so

So you may try to upgrade that library e.g. by following these commands as example:

$ locate libnss_nis-2.23.so /lib/x86_64-linux-gnu/libnss_nis-2.23.so $ dpkg -S /lib/x86_64-linux-gnu/libnss_nis-2.23.so 

Then depending to which affected library belongs to, try upgrading that. If that won't help, try searching or reporting the relevant bugs.

For newrelic-X.so, see: PHP Agent Segfaults.

---

Please note that PHP 5.6 and below are no longer supported. See: PHP Supported Versions page. For example PHP 5.6 branch support was until 19 Jan 2017, so it's recommended to upgrade your PHP to the recent stable branch.

Alternatively when you can't upgrade your versions, enabling and investigating core dumps with lis of backtraces could point you to the right direction of the root of the crashes.