Apache handshaking slow on prefork configuration, need config tuning

Question

My server, running Ubuntu 14/Apache2.4.7, has an htaccess redirect to force all requests to use HTTPS. Generally, it responds quickly. However, I recently altered my SSL config to restrict the ciphers used to more modern settings to prevent heartbleed, beast, and other exploits and ensure the use of modern encryption methods.

I've gotten some complaints lately -- which appear unrelated to these changes (see below) -- that my machine can be sluggish at times. These complaints suggest it might be related to handshaking or making a connection:

I've been getting intermittent slow loading of pages with various browser status messages depending on the browser used; all to the effect of waiting to establish a secure connection.

I wrote a PHP script that uses cURL to connect a thousand times. I ran it late last night and the slowness didn't seem to be an issue. I tried connecting to https and also to http and following the redirect above. All requests completed in 3.5 seconds or less, with the vast majority (96-98%) completing in less than 1.5 seconds.

I ran the same script at 10:30 this morning cali time and about 10% of the requests took longer than 1.5 seconds with many taking much more. The longest times were around 17 seconds.

My research and intuition tell me that, while https handshaking is more complex than http connections, this problem can probably be remedied by tweaking my apache configuration (e.g., MaxRequestWorker) settings. The server almost never exceeds a load average of 1.5 or so and it looks like there's plenty of memory available.

Can anyone suggest how I might narrow down the bottleneck here and what steps I might take to fix it? Any help would be much appreciated.

EDIT: I visited the #apache IRC channel and asked around there and the friendly folks drew my attention to the fact that this server is in prefork mode and the MinSpareServers, MaxSpareServers, StartServers, and MaxRequestWorkers are all either default or adjusted but still pretty low.

They were pretty adamant that a production server should not be using prefork mode and referred me to a few links:

• Apache MPM Event - describes apache event mode
• Running PHP on Apache httpd - a list of options to run php on apache
• A tale of openssl_seal(), PHP and Apache2handle - description of exploits on php of which to be aware

It's my understanding that the correction solution for these performance problems is probably to install apache to run in event mode, but my website is complex and uses some process forking and stuff. I'm hoping that as a stopgap solution, some can suggest tweaks to minSpareServers, maxSpareServers, startServers, maxRequestWorker

EDIT 2: A breakdown of a typical slow request, as reported by curl_getinfo function in PHP:

elapsed: 17.6722049713 ssl_verify_result: 0 total_time: 17.671187 namelookup_time: 0.000051 connect_time: 0.065855 pretransfer_time: 16.787012 starttransfer_time: 17.340403 redirect_time: 0.261569 

Answer 1

As it turns out, Ubuntu's packages to install apache and php7 do not appear to offer anything but prefork which really sucks. That being the case, I have had to make do with prefork which is not the ideal solution at all but until I have more info about getting one of the recommended configurations working, it's my only option.

I edited the file /etc/apache2/mods-available/mpm_prefork.conf:

sudo nano /etc/apache2/mods-available/mpm_prefork.conf 

And increased the MaxRequestWorkers setting to 300. NOTE that in order for me to do this, I have to also add a ServerLimit setting of 300 or MaxRequestWorkers is constrained to the default value of ServerLimit, which is 256:

StartServers 5 MinSpareServers 5 MaxSpareServers 10 # changed in response to slow connect/handshaking time complaints ServerLimit 300 # note this value is constrained by ServerLimit, which defaults to 256 MaxRequestWorkers 300 MaxConnectionsPerChild 0 

These settings appear to have alleviated the slowness. I used my PHP test script during a busy time of day and all requests completed in 3.5 seconds or less, the vast majority completing in under 1.5 seconds. I also tested with apache bench and a high concurrency value and I could see the server respond by spawning workers, but I still had plenty of free RAM:

ab -n 1000 -c 100 "https://example.com/" 

I checked RAM with:

free -h 

This appears to be working for now, although it is extremely disappointing that I can't get apache to use fastCGI using the ubuntu installers.