3

We've got an Apache server that almost daily becomes unresponsive. By checking /server-status (mod_status) we can see that we've got 60 child processes that are all in a "W" (Sending Reply) state.

  1. If we run service httpd restart everything goes back to normal and the problem goes away for a day or so.
  2. If we instead of restarting Apache kill every single child process, the problem remains (this is the only way for us to access /server-status which responds until all processes get to a "W" state).
  3. To me it seems that our PHP scripts never finish when the problem starts happening, which made me think it was a MySQL, Solr, or PHP/Apache timeout problem.
  4. However...
    • Solr/MySQL respond instantly.
    • There are plenty of MySQL connections available (we use AWS-RDS, the max connections allowed is greater than the number of Apache processes).
    • RAM is still fine (each process is 10m x 60 = 600Mb RAM, there's till plenty free).
    • PHP has max_exectution_time set to "30".
    • Apache TimeOut is set to "60".
    • We don't use persistent MySQL connections.
    • We do use curl_setopt($conn, CURLOPT_FORBID_REUSE, 0) to query Solr (I'm hoping this gets garbage collected properly by curl if the connection goes away).
  5. It still seems though that many processes never finish... I left a process running while killing all the other processes and this process stayed alive for 2 hours, still serving the exact same page (I could see this in /server-status) that normally take 50ms to respond.
  6. We don't use set_time_limit(0) or anything silly like that in our code.
  7. I assume that omitting set_time_limit means the scripts will finish after max_execution_time.

I had a theory that Apache's ListenBacklog as set too high and that whenever we killed the processes 60 new ones were instantly started, all trying to respond to clients that had long gone away. This would explain why the problem went away when we restart the server. But it seems ListenBacklog wasn't set and hence the default "511" would be in use. I tried killing all child processes several times in a row to flush the backlog, but the problem remains... all new requests to PHP pages take forever to respond (most don't respond).

PHP config:

max_execution_time = 30 max_input_time = 60 safe_mode = off

Apache config:

KeepAlive off <IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 60 MaxRequestsPerChild 1000 </IfModule>

I've run out of ideas... Any hints would be greatly appreciated!

Improve this question
1
  • I encountered what seems to be exactly this issue on aws, ubuntu12.04, php5.5, apache2.4(prefork+modphp). I ended up making similar adjustments. Specifically "ListenBacklog 1; Timeout 60;" lets apache recover in ~ 3min. Commented Jan 10, 2015 at 3:05

1 Answer 1

Reset to default
1

The troubleshooting steps I would recommend are:

  • strace -p $PID on a hung process to see what system calls, if any, it is stuck on
  • lsof -p $PID on that process to see if the filehandles or sockets that are open can give you a clue
  • tcpdump -vv -A -s1500 port 80 to see what the traffic is and where the response goes wrong.
Improve this answer
1
  • Thank you for your response! I tried the commands when the problem re-appeared but wasn't sure how to interpret the results. strace sayd "FUTEX_WAIT". lsof tells me "can't identify protocol". tcpdump spits out a lot of traffic, even though I can't access the server. For now I had to restart the server as I didn't have any other solution, but I will keep testing when the problem re-appears. Thanks again. Commented Jul 15, 2013 at 9:29

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.