Using `top` command shows PHP FPM processes using more memory than available

Question

I've read this answer about understanding top, as well as man top, but I think I still have trouble turning the presented data from top into actual information.

I'm logged on an Amazon EC2 instance which is part of a load-balanced group. It's running PHP FPM and I've filtered top to show only the php-fpm processes:

top - 11:27:43 up 18:59, 2 users, load average: 0.59, 0.79, 0.74 Tasks: 171 total, 1 running, 125 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 9.1 sy, 0.0 ni, 90.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 1935.6 total, 177.7 free, 692.3 used, 1065.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1052.0 avail Mem PID USER PR VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1566 root 20 356.2m 30.5m 23.5m S 1.6 0:02.14 php-fpm: master process (/etc/php-fpm.conf) 2188 webapp 20 442.2m 51.3m 31.8m S 2.7 2:18.47 php-fpm: pool www 2189 webapp 20 442.1m 50.8m 30.4m S 2.6 2:19.21 php-fpm: pool www 2190 webapp 20 444.2m 52.6m 30.4m S 2.7 2:20.14 php-fpm: pool www 2191 webapp 20 442.1m 51.1m 31.3m S 2.6 2:22.69 php-fpm: pool www 2192 webapp 20 442.2m 50.8m 31.6m S 2.6 2:19.64 php-fpm: pool www 2193 webapp 20 436.3m 46.5m 32.0m S 2.4 2:17.29 php-fpm: pool www 2194 webapp 20 452.2m 60.6m 30.4m S 3.1 2:19.82 php-fpm: pool www 2195 webapp 20 438.1m 48.0m 31.6m S 2.5 2:17.78 php-fpm: pool www 2197 webapp 20 442.2m 50.9m 30.6m S 2.6 2:18.28 php-fpm: pool www 12626 webapp 20 443.6m 50.4m 28.4m S 2.6 0:37.02 php-fpm: pool www 12627 webapp 20 443.5m 50.3m 28.6m S 2.6 0:35.68 php-fpm: pool www 12628 webapp 20 438.0m 45.1m 29.1m S 2.3 0:36.28 php-fpm: pool www 12629 webapp 20 443.9m 51.4m 29.3m S 2.7 0:35.26 php-fpm: pool www 12630 webapp 20 441.6m 48.9m 29.5m S 2.5 0:34.90 php-fpm: pool www 12631 webapp 20 443.6m 50.5m 28.7m S 2.6 0:34.93 php-fpm: pool www 12632 webapp 20 436.0m 43.2m 29.1m S 2.2 0:36.01 php-fpm: pool www 12635 webapp 20 441.6m 48.1m 28.3m S 2.5 0:34.56 php-fpm: pool www 12636 webapp 20 446.1m 55.0m 30.8m S 2.8 0:37.10 php-fpm: pool www 12637 webapp 20 441.9m 48.8m 29.0m S 2.5 0:35.16 php-fpm: pool www 12639 webapp 20 443.6m 50.3m 28.5m S 2.6 0:34.23 php-fpm: pool www 12640 webapp 20 438.0m 44.7m 28.9m S 2.3 0:36.33 php-fpm: pool www 12641 webapp 20 442.8m 49.5m 28.4m S 2.6 0:35.51 php-fpm: pool www 12642 webapp 20 443.8m 50.8m 29.1m S 2.6 0:36.22 php-fpm: pool www 12643 webapp 20 438.0m 44.2m 29.2m S 2.3 0:33.49 php-fpm: pool www 12644 webapp 20 440.0m 47.4m 29.3m S 2.5 0:36.44 php-fpm: pool www 12645 webapp 20 441.6m 48.7m 29.0m S 2.5 0:34.38 php-fpm: pool www 12646 webapp 20 441.6m 48.5m 28.8m S 2.5 0:34.53 php-fpm: pool www 12647 webapp 20 437.6m 44.5m 28.5m S 2.3 0:34.73 php-fpm: pool www 12648 webapp 20 437.7m 44.4m 28.6m S 2.3 0:33.64 php-fpm: pool www 12649 webapp 20 440.0m 46.9m 29.0m S 2.4 0:35.81 php-fpm: pool www 12651 webapp 20 444.1m 51.1m 29.0m S 2.6 0:34.77 php-fpm: pool www 12652 webapp 20 439.8m 47.0m 29.1m S 2.4 0:35.02 php-fpm: pool www 12657 webapp 20 443.9m 51.7m 29.9m S 2.7 0:35.35 php-fpm: pool www 12658 webapp 20 438.0m 45.5m 29.3m S 2.3 0:34.81 php-fpm: pool www 12667 webapp 20 441.9m 49.6m 29.6m S 2.6 0:34.21 php-fpm: pool www 

Here's what I don't understand:

1. What does the VIRT column display actually? I understand that it's virtual memory, but on line 2 of the memory summary, I can see that there's 1052.0 avail Mem, while the processes in the VIRT column sum up to way more than that.
2. How is there 1052.0 of available swap/virtual memory, while there's 0.0 total?
3. How is there only 692.3 memory used, when the sum of the %MEM column is 88.6%? Shouldn't it be at least 1935 * 0.886 = 1714? It looks like I get close to that number if I add in the 1065.7 buff/cache to those 692.3, but what's the reason behind that?
4. Why is the master process sleeping (as indicated by the S column)? Shouldn't at least it be running?

My main concern

The memory_limit setting in PHP ini is set to 256M. That should mean that each child process spawned by FPM could use up to that much memory, correct? If that's right and I have 35 child processes (as currently shown by top), then theoretically they could use (or attempt to use) up to 35 * 256 = 8960M memory. That's way more than the 1935M I have in total.

Also, the total sum of memory in the RES column is 1749.6. In other words, the used memory is 1749/1935 = 90.38%. But that's when all processes are in a sleep state! If there's a spike in traffic and memory consumption suddenly goes up, that sounds disastrous.

I think about reducing the allowed memory from 256M to 128M and reducing pm.max_children from the default 50 to, say, 12. This way, the total possible memory consumption of PHP FPM would be 12 * 128 = 1536M, which will be within the actual available memory on the system. Does that make sense?

Answer 1

A few things to consider. Firstly, you see 692.3 used, but you also need to factor in 1065.7 buff/cache. If you sum the RES column it should align with the total of used and buff/cache. This is because buff/cache represents data used for those processes, and while not immediately available, should be available shortly.

As to memory_limit and pm.max_children: You can run the most resource hungry script in your app and measure its memory usage using a memory_get_peak_usage().

Take that value, add in some capacity for growth of your dataset, and now you have a realistic maximum memory needed per script. With that figure you can now estimate the pm.max_children your server can handle. Adjust this figure down to allow for other processes on your server.

You can also now set a more accurate memory_limit - but be aware breaching this value will cause your script to die. I find it often works best as a control to stop runaway scripts from killing your service, and set it higher than my most memory demanding scripts to allow for future, more complex jobs.

You should also check the fpm (pm.*) variables and adjust accordingly.