21

Currently having a few issues with our server where, intermittently, we seem to get apache processes which just run and run, taking up 100% CPU.

When running top, we see the following: 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20788 www-data 20 0 318m 18m 3984 R 100 0.0 40:29.21 /usr/sbin/apache2 -k start 23523 www-data 20 0 319m 20m 4684 R 100 0.0 4:12.36 /usr/sbin/apache2 -k start

I want to try and find out what script (or whatever it is) is causing this, so I tried:

 strace -p 20788

But that doesn't show any output at all (I've left it for about 10 minutes, and it shows nothing). From my understanding, this could mean it's stuck in an infinite loop, and there aren't any "system calls" to show.

Is there anything else I can do to show what's going on?

Thanks

Edit - Forgot to mention, this is a live server with a few hundred users at any one time! So I can't really just freely try changing config options and restart apache.

Edit 2 - The backtrace (bt) from gdb doesn't seem to be all that useful when PHP isn't configured with --enable-debug - it only shows "execute()", but I need to know what PHP script is actually running.. is there any other way?

#0 0x00007f6c143fb0c5 in ?? () from /usr/lib/apache2/modules/libphp5.so #1 0x00007f6c143b040b in execute () from /usr/lib/apache2/modules/libphp5.so #2 0x00007f6c1438b970 in zend_execute_scripts () from /usr/lib/apache2/modules/libphp5.so #3 0x00007f6c14337fe3 in php_execute_script () from /usr/lib/apache2/modules/libphp5.so #4 0x00007f6c1441ae7d in ?? () from /usr/lib/apache2/modules/libphp5.so #5 0x00007f6c18912508 in ap_run_handler () #6 0x00007f6c1891297e in ap_invoke_handler () #7 0x00007f6c18922570 in ap_process_request () #8 0x00007f6c1891f398 in ?? () #9 0x00007f6c18918fa8 in ap_run_process_connection () #10 0x00007f6c189271d0 in ?? () #11 0x00007f6c1892793a in ?? () #12 0x00007f6c189284e7 in ap_mpm_run () #13 0x00007f6c188fd4a4 in main ()
Improve this question
6
  • 1
    Apache supports "graceful" restart, so why wouldn't you? Commented Mar 13, 2013 at 22:51
  • 1
    I think when we tried it previously, it couldn't restart gracefully because of the "stuck" apache processes... although that might be wrong, it was a while ago. Commented Mar 13, 2013 at 23:05
  • Another trick is to run another instance of apache on different port, redirecting new connections to it. Commented Mar 13, 2013 at 23:52
  • @BT643 I know it's an old thread.....I have the same problem....and i don't know what php script cause that. Did you figure it out how to see what php script is actually running ??? Commented Mar 18, 2020 at 8:17
  • @calin24 The "UPD" section in the accepted answer solved it for me in the end. I used mod_status with extended to find all the PHP files running. Commented Mar 18, 2020 at 14:23

6 Answers 6

Reset to default
10

Well, in case you're feeling brave:

gdb -p 20788

then issue bt to see the stack-frame, for e. g.

And BTW, there's also ltrace to mention — try it as well.

UPD.: well, ok, since now we have an idea that Apache is really running something, why wouldn't ya look at mod_status output — Extended one?

Improve this answer
7
  • gdb isn't installed :( will have to wait until I go back to work tomorrow to see if I can install it without causing any issues.. ltracedidn't show any output either. Commented Mar 13, 2013 at 19:28
  • Just added the results from the gdb bt into the initial post.. doesn't really tell me much at all! Commented Mar 14, 2013 at 11:41
  • Oh, glad to see I've suggested the right direction. ) Commented Mar 14, 2013 at 12:00
  • @BT643, see UPD. Commented Mar 14, 2013 at 12:03
  • 5
    Realised mod_status was already enabled by default, it was just limited to access from 127.0.0.1. I just logged in via SSH and piped the output to a file curl domain.com/server-status > randomfile.html - then viewed the file. Turned out it was an old developers code getting stuck in a loop (PHP file)! All sorted now. Thanks for the help :) Commented Mar 14, 2013 at 14:13
7

An very easy approach is to use htop. You can sort for the high CPU processes and then use

  • s for strace a process
  • l for lsof to see the open files of a processes
  • L to ltrace.

I found that at least one of that options finds the script that generates the load and you can of course use this on a production web server to debug.

Improve this answer
2

You could try:

  • iotop (showing I/O on the system)
  • netstat -t (showing connections)
  • Take a look at the apache logfiles and find out what the server did last
  • set some RLimits for the apache process. When these limits are reached the process will be killed, giving you some more information
Improve this answer
0

Your command should work provided you make an HTTP request that triggers that PID.

Maybe you want to temporarily re-configure Apache with only one child process?

Improve this answer
2
  • Keep in mind that only one child process means that Apache can only serve a single request, and if that single child is stuck, Apache will not be able to serve any requests. Commented Mar 13, 2013 at 18:48
  • Can't do that as it's a live server with hundreds of concurrent users (have added that into the OP as it wasn't clear before) Commented Mar 13, 2013 at 19:28
0

The PID of that apache instance is low, it might be the father of all the lot. That would certainly explain the high CPU usage (it stays around, others are spawned and recalled according to load). Much accumulated CPU time might just mean it has been running for a long time. No output from strace(1) just means it did no system calls. Yes, it might be in a tight loop, but apache is essentially I/O over the 'net, so I'd think it isn't doing anything useful. Strange 100% of one CPU, in any case.

Improve this answer
1
  • Low PID doesn't necessarily mean it's an old process. PIDs have a max value, and wrap around so new processes can be created using low PIDs. Commented Jul 7, 2015 at 13:13
0

Try this:

1) Start a log with date/time, PHP script, and the PID using getmypid()

2) Then watch your server with top

3) When you see the apache process going high, search for the same date/time and PID in your logs. You should be able to find the problematic script.

Improve this answer
1
  • This is an interesting solution but I can see it taking up more resources than it's worth, given that mod_status does its job quite well. Commented Jul 7, 2015 at 2:17

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.