Apache2 and nginx randomly consuming all memory, every week or so

Question

I'd like to preface this by saying I have read at least 10 related Serverfault questions before resorting to making my own...

I am currently running a Ubuntu 14.04.3 server with 2GB of RAM and about 5 active WordPress installations, all managed under the Vesta CP control panel.

Normally, it uses up about 700MB of the 2GB. But every week or so, all of the RAM becomes magically consumed, and the server slows down to almost a halt.

If I SSH into it and restart apache, as well as clear the memory (echo 3 > /proc/sys/vm/drop_caches), it starts functioning again just fine.

Here's my prefork module settings, which I feel are very reasonable:

<IfModule mpm_prefork_module> StartServers 5 MinSpareServers 1 MaxSpareServers 5 ServerLimit 10 MaxClients 10 MaxRequestsPerChild 1000 </IfModule> 

I even enabled mod_status and tried to see what PHP files were taking too long, but didn't find anything suspicious. Of course, when I look at the log while the server is down, it is flooded with at least 200 PHP files because they are unable to run due to the massive memory consumption.

I even enabled an 8GB SWAP file but that seems to have just delayed the inevitable.

Here's what the free -m command pulls up every time:

root@apache2-ps7881:/home/dhc-user# free -m total used free shared buffers cached Mem: 2001 1943 57 35 1 59 -/+ buffers/cache: 1883 118 Swap: 8191 4083 4108 

After restarting apache:

root@apache2-ps7881:/etc/apache2# free -m total used free shared buffers cached Mem: 2001 744 1257 65 36 204 -/+ buffers/cache: 503 1498 Swap: 8191 140 8051 

Here's the /var/log/apache2/error.log:

[Fri Feb 12 08:22:33.063204 2016] [mpm_prefork:error] [pid 2081] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting [Fri Feb 12 13:12:59.819680 2016] [core:warn] [pid 2081] AH00045: child process 6334 still did not exit, sending a SIGTERM 

That "child process still did not exit" error goes on for hundreds of more lines.

I get the server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting message every time it goes down.

Another error.log reveals the following:

[Fri Feb 12 08:19:55.781598 2016] [:error] [pid 20686] [client 10.10.10.9:54559] PHP Warning: mysqli_connect(): (08004/1040): Too many connections in /[censored]/$ [Fri Feb 12 08:19:55.896491 2016] [:error] [pid 20686] [client 10.10.10.9:54559] Too many connections, referer: http://[censored] 

Could it be that there's a connection not being closed? But would that cause a memory leak?

Here's an example of what the graphs reveal during the crashes:

Answer 1

I had a similar issue where a Wordpress site would start using tons of resources to the point where the server became largely unresponsive. Investigating the logs, I saw a few hundred attempts to access xmlrpc.php right around the same time as the memory footprint would balloon. The functions in xmlrpc.php can be abused as a force multiplier in brute force attacks using the system.multicall method.

This article is a more articulate description of how it works: https://blog.sucuri.net/2015/10/brute-force-amplification-attacks-against-wordpress-xmlrpc.html

More importantly, here are a few mitigation strategies from that article:

Protecting Yourself

I used to recommend people block all access to xmlrpc.php, but it was breaking some plugin’s functionality (mostly JetPack). With that in mind, if you are not using JetPack or any of the other plugin that require it XML-RPC, it might be a good idea to block direct access to it altogether.

If you can’t block XML-RPC, and you are using a WAF (web application firewall), I highly recommend blocking system.multicall requests. It is barely used in the wild and will protect you against these amplification methods.

I don't use any plugins that require access to xmlrpc.php, so I modified .htaccess to deny access. Since then, no more malicious actors successfully crushing the site. Here's the code if you'd like to give that a try:

Using the text editor of your choice, modify /var/www/html/.htaccess to include:

 <Files "xmlrpc.php"> Order Allow,Deny deny from all </Files> 

Wordpress has additional guidelines for hardening access to your site found here: https://wordpress.org/support/article/hardening-wordpress/

The Login Security Solution plugin for Wordpress may also help. I'd post the link, but I lack the reputation. Sorry!

Answer 2

It looks like the site simply gets busy, but you've said that's not the case. It's running out of server resources, so you need to perhaps limit the resources before it exhausts them.

I'd also look at the plugins Wordpress is using, and the widgets. Remove anything that's not 100% necessary, for a test. I've seen plugins and widgets do awful awful things to Wordpress websites. At the very least you should be using a caching plugin, W3 Total Cache is best, from memory.

If it was my server I'd change the underlying architecture of the server to use Nginx and HHVM, including page caching. I have four Wordpress installs and one moderately busy website running on an aws t2.micro, which is 10% of one Xeon core (burstable to one full core) and 1GB of memory, though I do also use the Amazon RDS (relational database service). My sites load VERY quickly. Note that this change isn't necessary, you can probably fix your config and it should work fine, but I don't know how.

I've written up a significant tutorial on setting up Wordpress on AWS with Nginx, HHVM, etc. You can read it here. You could quite easily set it all up alongside your current infrastructure, run Nginx on a different port, and even use the same webroot and database - but I'd create a read only database user while I was in testing. Once you're happy that it's working just swap the ports and you'll be running nginx, hhvm, and it should be more reliable. If not at least you might get some community support, as the configuration I suggest is fairly standard.

If your websites have many anonymous visitors you can server a large number of requests from the nginx page cache, which takes virtually no CPU, and doesn't hit PHP. My little instance can serve 1000 cached pages per second, sometimes more, not really sure what the bottleneck is but it does that on 2% CPU from the cache. It can only do 11 pages per second if it hits PHP, a massive difference which shows how slow executing code is. It'll make your site faster. It should make it faster for all visitors, HHVM is quite a bit faster than PHP5.

Answer 3

The configuration you've shown us is explicitly contradicted by your description of "at least 200 PHP files" and the output of top - which clearly contains more than 10 instances. That clearing the vfs cache appears to remove the symptoms is somewhat alarming - the only impact that should have is to slow the server down.

All these processes are blocked - and they are not working for the database. There looks to be some ugly file contention going on here too.

It looks like a combination of allowing too many connections combined with a blocking file lock. The latter might be a DOS attack, particularly if you are using the default session handler. If this is the case, it should be evident from your access logs.

There's no quick fix here, but your next step should be fixing the config so it actually works, and disabling all the WordPress plugins. These plugins vary greatly in quality.

Really we should be voting to close this as too broad.