Kill processes longer than 1 hour matching a pattern

Question

I am having an issue in a PHP application, where some processes seem to run for days - still trying to work out how/why but, whilst this happens, the resources on the server get zapped and loads goes way up. I then manually go in and kill the processes and everything is fine until it happens again

Therefore, as a stop gap, I want to run a cron to kill all php processes older than say 1 hour

When I get a list of processes it shows me below (I have removed all the ones which are not relevant)

 2049 7204994 php-fpm: master process (/etc/php-fpm.conf) 42146 42087 php-fpm: pool www -DFOREGROUND 47118 49511 php-fpm: pool www 47119 49511 php-fpm: pool www 47162 47222 php-fpm: pool www -- uid 0 -- gid 0 -- logger=files 47280 39490 php-fpm: pool www 51993 12066 php-fpm: pool www 51994 12066 php-fpm: pool www 53634 8575 php-fpm: pool www 53749 2590 php-fpm: pool www 53775 766 php-fpm: pool www 53780 447 php-fpm: pool www 53781 358 php-fpm: pool www 53784 111 php-fpm: pool www 53797 26 php-fpm: pool www 53799 0 ps -eo pid,etimes,cmd 

I want to be able to kill any process older than 3600 seconds BUT not including the master process or those with additional arguments

If I try

ps -eo pid,etimes,cmd | awk '$2 > 3600 && $3 == "php-fpm: pool www"' 

then nothing is returned at all.

If I try

ps -eo pid,etimes,cmd | awk -v pat="php-fpm" '$2 > 3600 && $3 ~ pat' 

this returns me all of the above which I do not want as I do not want to kill the master process but if I try

ps -eo pid,etimes,cmd | awk -v pat="php-fpm: pool" '$2 > 3600 && $3 ~ pat' 

OR

ps -eo pid,etimes,cmd | awk -v pat="pool" '$2 > 3600 && $3 ~ pat' 

then I get nothing returned at all

How can I get this to return only those where the pattern contains exactly "php-fpm: pool www" and nothing else?

Answer 1

still trying to work out how/why

Because this is exactly the job of the fast process manager.

The output you've shown us here is a set of mostly idle php worker processes. Your webserver submits requests to the master process (42146) which then checks if it has an idle process available to assign the request to. If not, and the defined limits will not be exceeded, it starts up a new process for the purpose.

Maintaining a pool of idle servers avoids latency waiting for a new process to start.

Actually FPM can start additional servers in advance of it running out of idle servers.

This is controlled by max_children, start_servers, max_spawn_rate min_spare_servers, max_spare_servers and process_idle_timeout

Killing processes is not the way to fix your problem

the resources on the server get zapped

Then 1) the limits in your FPM config are set too high 2) your workload exceeds the capacity of your hardware (more hardware is one solution, fixing your code is the other).

Answer 2

Regarding

awk '$2 > 3600 && $3 == "php-fpm: pool www"'

awk works on space-separated fields by default so none of the fields, including the 3rd field in your script, $3, can contain spaces. Here are the values of $3 for every line of your input so you can see that it never contains a space and so trying to match any regexp that contains a space can never work:

$ ps -eo pid,etimes,cmd | awk '{print $3}' php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: php-fpm: ps 

There's various ways to get the result you want, including testing the whole line instead of $3, e.g. using any awk:

$ ps -eo pid,etimes,cmd | awk '$2 > 3600 && /^ *[0-9]+ +[0-9]+ +php-fpm: pool www$/' 47118 49511 php-fpm: pool www 47119 49511 php-fpm: pool www 47280 39490 php-fpm: pool www 51993 12066 php-fpm: pool www 51994 12066 php-fpm: pool www 53634 8575 php-fpm: pool www 

That assumes the white space in your ps output is blanks (the ps on my machine doesn't support -o so I can't test it), if not just change each blank followed by a * or + in the regexp to [ \t] or [[:space:]] or similar to match any possible white space.