Apache: orphaned fcgi php processes

Question

We have a webserver with apache 2.2.14, PHP 5.3.2.

PHP is executed using mod_fcgid (see bottom). Everything works fine but sometimes, and we still have to figure out what triggers this, when php processes are "rotated" they remain active and orphaned: apache spawns new php processes and the old ones remains on the system. Killing them doesn't always kicks them away. More likely an "apache2ctl graceful" frees the system from this stale processes. We found this in error log: [Tue Jun 18 20:49:54 2013] [warn] mod_fcgid: process 2009 graceful kill fail, sending SIGKILL from what I found searching around this is quite normal, but that's all I found in apache logs during one of this processes leak.

This event is luckily seldomly happening, usually apache and php operate normally and with no problems during fcgid childrens renewal. How can we understand what's going wrong in these situations?

mod_fcgid config in site:

<IfModule mod_fcgid.c> SuexecUserGroup domain domain <Directory /var/www/fomain.it/htdocs/> AddHandler fcgid-script .php FCGIWrapper /var/www/fcgi/domain.it/fcgi-starter-php .php Options +ExecCGI -Indexes AllowOverride FileInfo Options Order allow,deny Allow from all </Directory> <Directory /var/www/fcgi/domain.it/> AllowOverride None Options +ExecCGI MultiViews -Indexes Order allow,deny Allow from all </Directory> </IfModule> 

/var/www/fcgi/domain.it/fcgi-starter-php:

#!/bin/sh PHPRC=/var/www/fcgi/domain.it/php/ export PHPRC PHP_FCGI_CHILDREN=8 export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=5000 export PHP_FCGI_MAX_REQUESTS exec /usr/lib/cgi-bin/php $1 

We currently found a workaround to avoid the server to fill up with orphaned processes. You can get rid of the stale processes with:

apache2ctl graceful 

and kill those processes with:

pkill -f -x /usr/lib/cgi-bin/php -P 1 

scripting and scheduling those two commands (with proper checks) will avoid the server to host plenty of useless procs, but the issue is still present.

Answer 1

I had a similar problem that fcgid ran out of available process slots after some period of activity.

The log message was roughly:

[fcgid:warn] mod_fcgid: can't apply process slot for /var/www/cgi-bin/xxx/php-cgi, referer: ... 

I tracked the issue down to this:

[fcgid:emerg] (35)Resource deadlock avoided: [client ....] mod_fcgid: can't get pipe mutex, referer: ... 

which gets caused by improper locking. In my case, Apache used fcntl() locking (by default on debian), so I changed that to flock() in apache2.conf:

Mutex flock:${APACHE_LOCK_DIR} default 

Reference that led me to the solution: https://bz.apache.org/bugzilla/show_bug.cgi?id=53999

Documentation about various locking options (fcgid has a warning advice against usign it with anything that would involve threads): https://httpd.apache.org/docs/2.4/mod/core.html#mutex

Answer 2

Changing to "sem" as Thomas suggested in https://bz.apache.org/bugzilla/show_bug.cgi?id=53999 worked for me.

So I would recommend anyone to edit his apache2.conf, comment the Mutex line, and put:

# Mutex file:${APACHE_LOCK_DIR} default # Original debian config Mutex sem # Solves orphaned PHP processes. 

Answer 3

I don`t have a real answer, but maybe more information can help to solve the problem. I like to tell, I have the same problem on a windows server with PHP 5.3.5.

Some cgi-processes remain as a kind of zombie task, after the done their real execution. They even ignore settings like max_execution time.

Currently I have a scheduled script what kills these old processes to. problem with this solution is, even "normal" cgi processes where killed, maybe it would be usefull to detect the runtime of the processes and only kill them if they have expired their max_execution time.