PHP + Apache long wait time

Question

I've run into a bit of a brick wall with troubleshooting my dedicated websever. Recently, my website spiked with the amount of request/per second and it crashed.

The box original had 8GB of RAM, 8 Core Xeon E3-1230, 1TB 7,200 RPM Disk (No Raid), 100Mbit dedicated networking.

After the spike, I increased the RAM to 24GB in order to support more concurrent users.

Apache seems to handle well, even with 3000 concurrent users, it will return HTML and static content very quickly (un-cached).

To further test the differences between Apache/HTML and Apache/PHP, I ran ab.

Both test.html and test.php have exactly the same static content, the PHP does not call any includes and does not connect with MySQL.

HTML Test

ab -n 500 -c 50 http://www.~~.com/test.html

Connection Times (ms) min mean[+/-sd] median max Connect: 252 375 190.3 276 1399 Processing: 254 354 121.5 282 657 Waiting: 253 353 121.4 280 653 Total: 510 730 231.7 573 1675 

PHP Test

ab -n 500 -c 50 http://www.~~~.com/test.php

Connect: 248 275 51.1 267 1316 Processing: 256 4167 6210.2 2262 41489 Waiting: 253 4166 6210.2 2262 41489 Total: 509 4442 6212.4 2523 41754 

Pingdom also reports a long wait time when accessing a PHP script.

I'm getting a similar result on WebPageTest.org, although better, the first time to byte is F:

 Load Time **First Byte** Start Render DOM Elements Time Requests Bytes In Time Requests Bytes In First View 2.061s **0.839s** 0.000s 55 2.061s 20 428 KB 2.061s 20 430 KB 

Here is my top results:

I/O Test

Under heavy load, the wa% can increase to 95% for a few milliseconds.

I ran iostat during load:

avg-cpu: %user %nice %system %iowait %steal %idle 8.37 0.00 5.18 0.56 0.00 85.88 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 45.50 3.00 48.00 136.00 748.00 17.33 3.05 59.76 2.53 12.90 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb3 0.00 45.50 3.00 48.00 136.00 748.00 17.33 3.05 59.76 2.53 12.90 avg-cpu: %user %nice %system %iowait %steal %idle 4.00 0.00 3.56 0.69 0.00 91.75 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 6.00 118.50 9.50 21.00 996.00 1116.00 69.25 0.29 9.44 1.66 5.05 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb3 6.00 118.50 9.50 21.00 996.00 1116.00 69.25 0.29 9.44 1.66 5.05 

To me, that doesn't look bad, but I may be missing something.

I'm using FCGI

<IfModule mod_fcgid.c> FcgidMaxRequestLen 1547483648 FcgidMaxRequestInMem 52485760 FcgidIdleScanInterval 15000 FcgidBusyTimeout 15000 FcgidProcessLifeTime 7200 FcgidConnectTimeout 1800 FcgidIOTimeout 1800 PHP_Fix_Pathinfo_Enable 1 FcgidMaxRequestsPerProcess 1000 </IfModule> 

And here is my Apache Conf (I'm using 2.4.x)

Timeout 60 TraceEnable Off ServerSignature Off ServerTokens ProductOnly FileETag None StartServers 10 <IfModule prefork.c> MinSpareServers 5 MaxSpareServers 15 </IfModule> <IfModule itk.c> MinSpareServers 5 MaxSpareServers 15 </IfModule> ServerLimit 2200 MaxRequestWorkers 2000 MaxConnectionsPerChild 15000 KeepAlive On KeepAliveTimeout 1 MaxKeepAliveRequests 2000 

I've looked into my Apache Error Logs and Access Logs. Nothing strange to report.

I'm really scratching my head here.

I've tried turning off the firewall.

I've tried increasing Max Connections.

I've optimized mySQL and removing many slow queries (that were >0.5s).

What else can I do, is there anything I can use to help identify problems? Any help would be greatly appreciated.

P.S:

It's worth noting, that even when the server is being heavily accessed, PHPMyAdmin and cPanel are still very responsive. Nothing else seems to lag except the PHP on the website.

Answer 1

I think your problem is simply your hard disk with it's access times. Apache can cache html-pages in memory, but php-scripts are not cached; the need to be executed every time again. Therefore the php interpreter is called and it reads the script from the hdd. This takes a lot of time. The most slowest thing on your server is your HDD. On my pc, there is an extreme difference between applications starting from my SSD and my HDD (the SSD is up to 10 times faster!). If your HDD works, then the latency for executing an php script may increase drastically.

Possible solutions: get an SSD (maybe a small one just for often accessed data such as scripts) and minimize script calls (and HDD accesses) on your server. Make sure, the filesystem is defragmented (usually done automatically). If your php-script creates often the same contents, try caching them into an html file.

This answer could also help you: https://stackoverflow.com/questions/4181865/apache-php-caching

Answer 2

HTML file you're testing is just a plain file, all that Apache has to do is do a few system calls (open, read) and then serve its content.

PHP OTOH is actually quite a "heavy" option: it's an entire interpreter (bytecode compiler?). And since you're using concurrent testing (-c) who knows how well the requests are multiplexed to it? This does not have to be apache problem at all, but rather PHP's problem.

What I'd do:

1. Switch to Apache MPM (multi-threaded in multiple processes).

2. Do sequential test (no multiple concurrent requests), compare.

3. Run valgrind or some such on apache process and see where most CPU time is spent (apache or PHP).

4. run the same test but serving this page via nginx. Since nginx is based on async model and very fast, then if you get similar results, PHP is the culprit.

5. Finally, you could install Zend Optimizer (30-day trial is free) or smth like this: https://github.com/zendtech/ZendOptimizerPlus.

Really, comparing serving a static file to dynamically generated webpage is sort of apples to oranges comparison. None of the typical solutions (Python mod-apache, Django, PHP, etc) are going to be very fast in this regard, at least when compared to serving a static file. Node.js is an exception perhaps, due to sort of "low level" programming webpage directly in async model.

P.S. you have not quoted php.ini content. Post it and/or tweak it.