php crashes with no core file and this message : apc_mmap failed

Question

Description of the problem

Regularly, cron php processes crash on our production server, which result in mails with the following body :

PHP Fatal error: PHP Startup: apc_mmap: mmap failed: in Unknown on line 0 Segmentation fault (core dumped)

I think the Segmentation fault (core dumped) should result in core files being handled by apport and then written in /var/crashes, but the files I can see there are there since yesterday, although the last crash occured today :

-rw-r----- 1 root whoopsie 1138528 mai 22 04:09 _usr_bin_php5.0.crash -rw-r----- 1 frontoffice whoopsie 1166373 mai 20 18:00 _usr_bin_php5.1005.crash -rw-r----- 1 frontoffice whoopsie 81622658 mai 22 00:05 _usr_sbin_php5-fpm.1005.crash 

I tried to download the last one anyway, and ran gdb /usr/sbin/php5-fpm /tmp/_usr_sbin_php5-fpm.1005.crash, only to be told that the file is not a core file (its format was not recognized).

Here is the server's apc configuration :

cat /etc/php5/cli/conf.d/20-apc.ini extension=apc.so apc.shm_size=512M apc.ttl=3600 apc.user_ttl=3600 apc.enable_cli=1 

I'm mostly worried about the apc.shm_size… isn't it too high or too low ? I understand it has to do with the size of memory segments.

Question(s)

1. What could be the problem ?
2. How can I troubleshoot it (how can I get a valid core file ?) ?

System information

free total used free shared buffers cached Mem: 5081296 4354684 726612 0 374744 959968 -/+ buffers/cache: 3019972 2061324 Swap: 522236 516888 5348 cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS" php -v PHP 5.4.17-1~precise+1 (cli) (built: Jul 17 2013 18:14:06) Copyright (c) 1997-2013 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies 

php -i excerpt :

Configuration apc APC Support => enabled Version => 3.1.13 APC Debugging => Disabled MMAP Support => Enabled MMAP File Mask => Locking type => pthread mutex Locks Serialization Support => php Revision => $Revision: 327136 $ Build Date => Nov 20 2012 18:41:36 Directive => Local Value => Master Value apc.cache_by_default => On => On apc.canonicalize => On => On apc.coredump_unmap => Off => Off apc.enable_cli => On => On apc.enabled => On => On apc.file_md5 => Off => Off apc.file_update_protection => 2 => 2 apc.filters => no value => no value apc.gc_ttl => 3600 => 3600 apc.include_once_override => Off => Off apc.lazy_classes => Off => Off apc.lazy_functions => Off => Off apc.max_file_size => 1M => 1M apc.mmap_file_mask => no value => no value apc.num_files_hint => 1000 => 1000 apc.preload_path => no value => no value apc.report_autofilter => Off => Off apc.rfc1867 => Off => Off apc.rfc1867_freq => 0 => 0 apc.rfc1867_name => APC_UPLOAD_PROGRESS => APC_UPLOAD_PROGRESS apc.rfc1867_prefix => upload_ => upload_ apc.rfc1867_ttl => 3600 => 3600 apc.serializer => default => default apc.shm_segments => 1 => 1 apc.shm_size => 512M => 512M apc.shm_strings_buffer => 4M => 4M apc.slam_defense => On => On apc.stat => On => On apc.stat_ctime => Off => Off apc.ttl => 3600 => 3600 apc.use_request_time => On => On apc.user_entries_hint => 4096 => 4096 apc.user_ttl => 3600 => 3600 apc.write_lock => On => On php -m [PHP Modules] apc bcmath bz2 calendar Core ctype curl date dba dom ereg exif fileinfo filter ftp gd gettext hash iconv imagick intl json ldap libxml mbstring memcache memcached mhash mysql mysqli openssl pcntl pcre PDO pdo_mysql pdo_pgsql pdo_sqlite pgsql Phar posix Reflection session shmop SimpleXML soap sockets SPL sqlite3 standard sysvmsg sysvsem sysvshm tidy tokenizer wddx xml xmlreader xmlwriter zip zlib [Zend Modules] ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 39531 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 39531 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

Answer 1

The core file needs to be read on a system that is at least very similar to the one where the crash happened. In particular you need to have the same versions of the binary and all involved libraries in order for the pointers to line up. Usually it's easiest to run gdb on the machine where the crash happened. You'll also need to have versions of the binary and libraries installed that have the symbolic data you need to identify locations in the source files where things happened. That might mean the dev versions of the various libraries, but It depends what distribution of linux you run.

Are you sure you have the right version of APC installed? Eg it solved this person's problem: https://stackoverflow.com/questions/14756385/php-fatal-error-php-startup-apc-mmap-mmap-failed-in-unknown-on-line-0

Is APC failing for web processes as well as command line ones? If it only fails for one of those, then check that both php packages are the correct versions to work with your version of APC.

The first two dump files you listed look very small to me. Just over 1 MB. PHP would usually get bigger than that before it gets as far as running any of your code. That's likely consistent with failing before loading the code though, and given APC is involved, that's likely. The fpm one is a web process, not a cron job (unless your cron calls php via the web interface?)

Setting apc.shm_size to 512MB may or not be optimal for efficiency, but I wouldn't expect it to be the cause of a segfault. Corrupt data in your APC cache could conceivably be the problem though, so I suggest you clear the cache. The normal process is to use an apc.php file which is likely distributed with apc. Vendor distributions vary on that, but it is included with the upstream source code, so you should be able to get a copy easily enough. That gives you a web interface for looking at the state of your cache, and for clearing it. If APC is failing to the point where that doesn't work, I'm not sure what the process is. Probably locate the cache, delete it, and reinstall APC if needed to rebuild it. (kinda dirty approach, but low effort if you can afford a brief outage).

Answer 2

This should really be a comment, but it's a bit long

isn't it too high or too low ?

If you don't know how, then how should we? You haven't told us how much RAM and swap there is, how much is used for other stuff. You haven't told us how much of the APC memory is used before the system crashes.

file is not a core file (its format was not recognized).

Have you checked the ulimit? Most likely the file has been truncated. Regardless, a segmentation fault suggests an issue within PHP itself (or APC, or an extension). Were you planning on fixing it yourself? Don't get me wrong - the guys who write the stuff will welcome well researched and documented bug reports - but the first thing you should be looking at (and including in your question here) is the version of PHP, the extensions installed and the version of APC.