2

So I think my server might be suffering a Denial of Service attack.

We got notified by pingdom (website monitoring) that our website was unavailable starting around 3AM. Early today we started checking apache error logs and saw a whole bunch of this error:

AH00485: scoreboard is full, not at MaxRequestWorkers

We also saw that our PHP-FPM process pool frequently needed to spawn more servers:

[pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children

We tried increasing MaxRequestWorkers in the apache conf and some other remedies but these would not rid us of the scoreboard error in the apache error log so, against my better judgement, I followed the advice in this thread and set MinSpareThreads and MaxSpareThreads equal to MaxRequestWorkers. These changes appear to have removed the scoreboard error.

I also greatly increased MaxRequestWorkers because we have a lot of RAM that evidently isn't being utilized. Our server has 8 cores and, despite these really high config values, doesn't seem to be using much of its RAM at all:

$ free -h total used free shared buff/cache available Mem: 7.8G 1.8G 2.0G 38M 4.0G 5.8G Swap: 0B 0B 0B

I'm pretty nervous about these high values for MaxRequestWorkers in the apache conf and pm.max_children in php-fpm configuration.

Here's the basic config in mpm_event.conf

<IfModule mpm_event_module> StartServers 2 MinSpareThreads 800 MaxSpareThreads 800 ThreadLimit 64 ThreadsPerChild 25 ServerLimit 800 MaxRequestWorkers 800 MaxConnectionsPerChild 0 </IfModule>

Here are some settings in a php-fpm conf file:

pm.max_children = 256 pm.start_servers = 64 pm.min_spare_servers = 64 pm.max_spare_servers = 128

Here's some basic server info:

Server version: Apache/2.4.18 (Ubuntu) Server built: 2019-10-08T13:31:25 Server's Module Magic Number: 20120211:52 Server loaded: APR 1.5.2, APR-UTIL 1.5.4 Compiled using: APR 1.5.2, APR-UTIL 1.5.4 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count)

And here's some of the data from the apache server-status output:

Server Version: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g Server MPM: event Server Built: 2019-10-08T13:31:25 Current Time: Friday, 10-Jan-2020 22:58:55 CST Restart Time: Friday, 10-Jan-2020 22:26:32 CST Parent Server Config. Generation: 1 Parent Server MPM Generation: 0 Server uptime: 32 minutes 22 seconds Server load: 4.69 5.06 5.12 Total accesses: 78434 - Total Traffic: 1.5 GB CPU Usage: u2970.53 s5037.34 cu0 cs0 - 412% CPU load 40.4 requests/sec - 0.8 MB/second - 19.7 kB/request 797 requests currently being processed, 3 idle workers PID Connections Threads Async connections total accepting busy idle writing keep-alive closing 6124 28 yes 25 0 0 0 3 6125 27 yes 25 0 0 0 2 6182 30 yes 25 0 0 1 4 6210 28 yes 25 0 0 0 3 6211 29 yes 25 0 0 0 5 6266 28 yes 25 0 0 2 1 6267 25 yes 25 0 0 0 1 6269 28 no 24 1 0 1 3 6276 28 yes 25 0 0 0 3 6378 28 yes 25 0 0 0 3 6379 31 no 24 1 0 4 3 6380 27 yes 25 0 0 0 3 6384 26 yes 25 0 0 0 2 6397 28 yes 25 0 0 2 1 6405 27 yes 25 0 0 0 2 6414 26 yes 25 0 0 1 0 6423 27 no 24 1 0 1 1 6602 27 yes 25 0 0 0 3 6603 28 yes 25 0 0 0 4 6604 26 yes 25 0 0 0 1 6617 30 yes 25 0 0 0 5 6646 26 yes 25 0 0 0 2 6676 27 yes 25 0 0 0 2 6694 30 yes 25 0 0 0 5 6705 28 yes 25 0 0 0 3 6730 29 yes 25 0 0 0 4 6765 29 yes 25 0 0 0 4 6781 27 yes 25 0 0 0 2 6805 28 yes 25 0 0 0 4 6836 28 yes 25 0 0 0 3 6858 27 yes 25 0 0 0 3 6859 27 no 25 0 0 1 1 Sum 888 797 3 0 13 86

The worker mode part is the most disconcerting. Almost every single one is in read mode:

RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRR_RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR _RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR_RRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR

And at the end there's this:

SSL/TLS Session Cache Status: cache type: SHMCB, shared memory: 512000 bytes, current entries: 2176 subcaches: 32, indexes per subcache: 88 time left on oldest entries' objects: avg: 220 seconds, (range: 197...243) index usage: 77%, cache usage: 99% total entries stored since starting: 60122 total entries replaced since starting: 0 total entries expired since starting: 0 total (pre-expiry) entries scrolled out of the cache: 57946 total retrieves since starting: 3405 hit, 59594 miss total removes since starting: 0 hit, 0 miss

And netstat shows some 3000+ connections to port 80 and port 443:

$ netstat -n | egrep ":80|443" | wc -l 3715

What the heck is going on? The server has been running fine for months with much more modest configuration settings. Something seems to have abruptly changed last night around 3AM.

Any guidance would be much appreciated. I searched here first and found this other thread but it's a different version of apache running in prefork mode instead of event like mine. I also don't understand how the little bit of information in that thread led to a SlowLoris diagnosis.

EDIT It would appear I have to phrase my questions more precisely:

1) How can I restore my server's responsiveness. Clearly, the apache workers getting stuck in R mode is a sympton of some problem.

2) Is there some reliable series of steps I can take to more specifically identify the actual problem?

3) Is there any way to confirm that the machine is under a DoS attack?

Improve this question
3
  • call the police? Commented Jan 11, 2020 at 5:42
  • 1
    egrep ":80|443" should be egrep ":80|:443" Commented Jan 11, 2020 at 5:43
  • @poige thanks for egrep tip -- it's still quite high (3451 at the moment). I'd love to call the police, but it's been my experience that they really a) don't care, b) don't have the technical understanding or resources to deal with the issue and c) even when you spoon feed them the information they still have no idea what's going on. Commented Jan 11, 2020 at 5:47

2 Answers 2

Reset to default
1

Merely counting the number of connections on the scoreboard is not enough evidence to know clients are being rude and not following up on their connections. That is a drastic increase, so either the web app got very popular, or someone is making silly requests.

Look at the rate of requests finished per second. Should be pretty high with that many workers, assuming your web app is performing adequately. Check all aspects of the web server performance, including available bandwidth to users, server load, and the performance of related components like any databases. Fix any performance problems due to insufficient resources.

Do an analysis of the distribution of IP addresses connected the web ports. One IP doing all the hundreds of connections is unusual, although IPv4 NATs complicate this. Determine the ISPs of the source addresses. Check the IP addresses's security reputation scores, and if it could be an enormous NAT.

Do a packet capture on incoming requests, while still doing your monitoring. You should see at least some HTTP requests from well-behaved clients. If clients just connect and sit there, that looks a bit like SlowLoris style resource exhaustion.

Consider the tuning recommendations in the linked answer. On Linux, reducing the timeouts a bit with sysctl net.ipv4.tcp_fin_timeout = 10 or so is something to try.

Consider putting this web server behind a security-oriented and load balancing proxy. Web application firewall features could allow you to do clever things to filter requests. Scaling horizontally could enable you to handle more requests.

Improve this answer
0

Is there any way to confirm that the machine is under a DoS attack?

DoS is Denial of Service.

Attack is hostile action carried out to do harm.

(Passive aggression is an oxymoron used by people who don't understand that passive means absence of an action — inaction, by definition, and aggression (by definition too) means hostile action. But that's kinda another story, of course.)

Between those twos there's a gap where it's DoS but it's not an attack in terms of hostile action. Say, F5 gone stuck on a user's keyboard can cause DoS unless countermeasures are taken but it's not an attack as a hostile action carried out with intention to do harm. OTOH, it is an attack if user knows this would cause DoS and intentionally holds that key pressed.

So answering your question — it's obviously impossible to tell for sure unless you can prove there's an intention. It's possible to tell if it is a DoS if service interruption occurs due to lack of resources — overloading.

Improve this answer
5
  • 1
    While I appreciate your attempt to answer here, I think I wasn't entirely clear in my question. It is my primary desire to get my web server responsive again. It's painfully slow even though the server does not appear to be working very hard. Your answer is quite a bit more philosophical than what I was looking for. I've edited my question to hopefully make my needs more apparent. Commented Jan 11, 2020 at 6:09
  • Right, now it's somewhat different in nutshell. As to Apache in "R": serverfault.com/questions/503657/… Commented Jan 11, 2020 at 6:12
  • One answer there mentions Nginx among others, which is popular HTTP-frontend solution although it's not the only such a proxy. But at least Nginx has built-in mechanisms of requests limiting which can be helpful under circumstances similar to those you're facing. Commented Jan 11, 2020 at 6:16
  • I linked that very thread in my original post. Also, not sure what you mean about Nginx. I'm not running Nginx. I'm running apache. Am I supposed to rebuild my web app? Do Nginx and apache somehow work together? So many questions. Commented Jan 11, 2020 at 6:18
  • Right. It's obvious you're not running Nginx and that's can be a way to go. Nginx is very light-weight and performant "reverse proxy" meaning you can put it to handle HTTP traffic before it would dispatch it further to Apache. No, you won't have to rebuild your app. More likely you could get rid of Apache altogether but it's not required to bring Nginx in and have its perks. Commented Jan 11, 2020 at 6:23

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.