2

I have a VPS with standard LAMP stack and a busy website. Operating system is CentOS 5.5. Virtualization is done with VMWare. My server gets real slow about every 6 hours. Logging into it I see that 1.6GB of RAM is consumed. However, summing up the memory usage of active processes adds up only to about 700MB. Can anyone make any sense of this?

"free" shows this:

 total used free shared buffers cached Mem: 2059456 2049280 10176 0 14780 380968 -/+ buffers/cache: 1653532 405924 Swap: 2096472 96 2096376

While this is output of "ps":

[root@vmi29 /]# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 10348 688 ? Rs Jun05 0:01 init [3] root 2 0.0 0.0 0 0 ? S< Jun05 0:00 [migration/0] root 3 0.0 0.0 0 0 ? SN Jun05 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S< Jun05 0:00 [migration/1] root 5 0.0 0.0 0 0 ? SN Jun05 0:00 [ksoftirqd/1] root 6 0.0 0.0 0 0 ? S< Jun05 0:00 [migration/2] root 7 0.0 0.0 0 0 ? SN Jun05 0:00 [ksoftirqd/2] root 8 0.0 0.0 0 0 ? S< Jun05 0:00 [migration/3] root 9 0.0 0.0 0 0 ? SN Jun05 0:00 [ksoftirqd/3] root 10 0.0 0.0 0 0 ? S< Jun05 0:06 [events/0] root 11 0.0 0.0 0 0 ? S< Jun05 0:00 [events/1] root 12 0.0 0.0 0 0 ? S< Jun05 0:00 [events/2] root 13 0.0 0.0 0 0 ? S< Jun05 0:00 [events/3] root 14 0.0 0.0 0 0 ? S< Jun05 0:00 [khelper] root 31 0.0 0.0 0 0 ? S< Jun05 0:00 [kthread] root 38 0.0 0.0 0 0 ? S< Jun05 0:00 [kblockd/0] root 39 0.0 0.0 0 0 ? S< Jun05 0:00 [kblockd/1] root 40 0.0 0.0 0 0 ? S< Jun05 0:00 [kblockd/2] root 41 0.0 0.0 0 0 ? S< Jun05 0:00 [kblockd/3] root 42 0.0 0.0 0 0 ? S< Jun05 0:00 [kacpid] root 204 0.0 0.0 0 0 ? S< Jun05 0:00 [cqueue/0] root 205 0.0 0.0 0 0 ? S< Jun05 0:00 [cqueue/1] root 206 0.0 0.0 0 0 ? S< Jun05 0:00 [cqueue/2] root 207 0.0 0.0 0 0 ? S< Jun05 0:00 [cqueue/3] root 210 0.0 0.0 0 0 ? S< Jun05 0:00 [khubd] root 212 0.0 0.0 0 0 ? S< Jun05 0:00 [kseriod] root 302 0.0 0.0 0 0 ? S Jun05 0:00 [khungtaskd] root 303 0.0 0.0 0 0 ? S Jun05 0:00 [pdflush] root 304 0.0 0.0 0 0 ? S Jun05 0:01 [pdflush] root 305 0.0 0.0 0 0 ? S< Jun05 0:05 [kswapd0] root 306 0.0 0.0 0 0 ? S< Jun05 0:00 [aio/0] root 307 0.0 0.0 0 0 ? S< Jun05 0:00 [aio/1] root 308 0.0 0.0 0 0 ? S< Jun05 0:00 [aio/2] root 309 0.0 0.0 0 0 ? S< Jun05 0:00 [aio/3] root 515 0.0 0.0 0 0 ? S< Jun05 0:00 [kpsmoused] root 582 0.0 0.0 0 0 ? S< Jun05 0:00 [mpt_poll_0] root 583 0.0 0.0 0 0 ? S< Jun05 0:00 [mpt/0] root 584 0.0 0.0 0 0 ? S< Jun05 0:00 [scsi_eh_0] root 590 0.0 0.0 0 0 ? S< Jun05 0:00 [ata/0] root 591 0.0 0.0 0 0 ? S< Jun05 0:00 [ata/1] root 592 0.0 0.0 0 0 ? S< Jun05 0:00 [ata/2] root 593 0.0 0.0 0 0 ? S< Jun05 0:00 [ata/3] root 594 0.0 0.0 0 0 ? S< Jun05 0:00 [ata_aux] root 610 0.0 0.0 0 0 ? S< Jun05 0:00 [kstriped] root 631 0.0 0.0 0 0 ? S< Jun05 0:05 [kjournald] root 656 0.0 0.0 0 0 ? S< Jun05 0:00 [kauditd] root 689 0.0 0.0 13364 928 ? S<s Jun05 0:00 /sbin/udevd -d root 2123 0.0 0.0 0 0 ? S< Jun05 0:00 [kmpathd/0] root 2124 0.0 0.0 0 0 ? S< Jun05 0:00 [kmpathd/1] root 2126 0.0 0.0 0 0 ? S< Jun05 0:00 [kmpathd/2] root 2127 0.0 0.0 0 0 ? S< Jun05 0:00 [kmpathd/3] root 2128 0.0 0.0 0 0 ? S< Jun05 0:00 [kmpath_handlerd] root 2203 0.0 0.0 0 0 ? S< Jun05 0:00 [kjournald] root 2613 0.0 0.0 5908 648 ? Ss Jun05 0:00 syslogd -m 0 root 2617 0.0 0.0 3804 424 ? Ss Jun05 0:00 klogd -x root 2707 0.0 0.0 10760 372 ? Ss Jun05 0:02 irqbalance apache 2910 0.5 0.6 213964 12912 ? S 00:22 0:07 /usr/sbin/httpd dbus 3011 0.0 0.0 21256 904 ? Ss Jun05 0:00 dbus-daemon --system root 3025 0.0 0.0 3800 576 ? Ss Jun05 0:00 /usr/sbin/acpid 68 3038 0.0 0.2 31152 4336 ? Ss Jun05 0:01 hald root 3039 0.0 0.0 21692 1176 ? S Jun05 0:00 hald-runner 68 3046 0.0 0.0 12324 856 ? S Jun05 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.s 68 3052 0.0 0.0 12324 856 ? S Jun05 0:00 hald-addon-keyboard: listening on /dev/input/event0 root 3105 0.0 0.0 62624 1212 ? Ss Jun05 0:00 /usr/sbin/sshd root 3264 0.0 0.0 74820 1156 ? Ss Jun05 0:00 crond root 3292 0.0 0.0 18416 472 ? S Jun05 0:00 /usr/sbin/smartd -q never root 3300 0.0 0.0 3792 480 tty2 Ss+ Jun05 0:00 /sbin/mingetty tty2 root 3301 0.0 0.0 3792 480 tty3 Ss+ Jun05 0:00 /sbin/mingetty tty3 root 3302 0.0 0.0 3792 484 tty4 Ss+ Jun05 0:00 /sbin/mingetty tty4 root 3304 0.0 0.0 3792 480 tty5 Ss+ Jun05 0:00 /sbin/mingetty tty5 root 3306 0.0 0.0 3792 480 tty6 Ss+ Jun05 0:00 /sbin/mingetty tty6 apache 5158 0.4 0.5 211896 11848 ? S 00:28 0:04 /usr/sbin/httpd apache 5519 0.4 0.5 211896 11992 ? S 00:29 0:03 /usr/sbin/httpd root 5649 0.0 0.0 63848 1184 pts/0 S Jun05 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --soc mysql 5696 2.1 1.9 411060 40392 pts/0 Rl Jun05 2:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql apache 5943 0.4 0.5 211896 12000 ? S 00:30 0:03 /usr/sbin/httpd apache 5976 0.6 0.5 211896 11792 ? S 00:30 0:04 /usr/sbin/httpd apache 6073 0.4 0.5 211896 11208 ? S 00:31 0:03 /usr/sbin/httpd apache 6122 0.4 0.5 211896 11848 ? S 00:31 0:03 /usr/sbin/httpd apache 6128 0.3 0.5 211896 11940 ? S 00:31 0:02 /usr/sbin/httpd apache 6159 0.5 0.5 211896 11872 ? S 00:31 0:04 /usr/sbin/httpd apache 6636 0.4 0.6 213960 13444 ? S 00:32 0:02 /usr/sbin/httpd apache 6787 0.3 0.5 211884 11308 ? S 00:33 0:02 /usr/sbin/httpd apache 6796 0.4 0.5 211884 12024 ? S 00:33 0:02 /usr/sbin/httpd apache 6801 0.3 0.5 211896 11920 ? S 00:33 0:01 /usr/sbin/httpd apache 6804 0.4 0.5 211884 11848 ? S 00:33 0:02 /usr/sbin/httpd apache 6825 0.4 0.5 211896 11972 ? S 00:33 0:02 /usr/sbin/httpd apache 6866 0.3 0.5 210860 11044 ? S 00:33 0:01 /usr/sbin/httpd apache 6870 0.2 0.5 211896 11108 ? S 00:33 0:01 /usr/sbin/httpd apache 6872 0.3 0.5 211896 11900 ? S 00:33 0:01 /usr/sbin/httpd apache 6993 0.3 0.5 211896 11836 ? S 00:33 0:02 /usr/sbin/httpd apache 6994 0.3 0.5 211896 11792 ? S 00:33 0:01 /usr/sbin/httpd apache 7136 0.2 0.5 211896 11432 ? S 00:34 0:01 /usr/sbin/httpd apache 7143 0.2 0.5 210860 11052 ? S 00:34 0:01 /usr/sbin/httpd apache 7145 0.2 0.5 211896 11136 ? S 00:34 0:01 /usr/sbin/httpd apache 7266 0.2 0.6 213952 12748 ? S 00:34 0:01 /usr/sbin/httpd apache 7299 0.2 0.5 211884 11276 ? S 00:34 0:01 /usr/sbin/httpd apache 7311 0.2 0.5 211884 11300 ? S 00:34 0:01 /usr/sbin/httpd apache 7313 0.3 0.5 211884 11876 ? S 00:34 0:01 /usr/sbin/httpd apache 7345 0.2 0.5 210872 11100 ? S 00:34 0:01 /usr/sbin/httpd apache 7349 0.2 0.5 210860 11008 ? S 00:34 0:01 /usr/sbin/httpd apache 7350 0.2 0.5 211896 11832 ? S 00:34 0:01 /usr/sbin/httpd apache 7351 0.1 0.5 211884 11072 ? S 00:34 0:00 /usr/sbin/httpd apache 7352 0.2 0.5 210872 11096 ? S 00:34 0:01 /usr/sbin/httpd apache 7449 0.1 0.5 210860 11020 ? S 00:35 0:00 /usr/sbin/httpd root 7490 0.3 0.0 0 0 ? S Jun05 3:11 [vmmemctl] root 7597 0.0 0.0 72656 1260 ? Ss Jun05 0:06 /usr/lib/vmware-tools/sbin64/vmware-guestd --background /va apache 7720 0.1 0.5 210860 10748 ? S 00:36 0:00 /usr/sbin/httpd apache 7726 0.1 0.4 209836 9304 ? R 00:36 0:00 /usr/sbin/httpd apache 7727 0.1 0.5 210860 10916 ? S 00:36 0:00 /usr/sbin/httpd apache 7731 0.1 0.5 210860 10780 ? S 00:36 0:00 /usr/sbin/httpd apache 7732 0.3 0.5 210860 10916 ? S 00:36 0:01 /usr/sbin/httpd apache 7733 0.1 0.5 210872 11000 ? S 00:36 0:00 /usr/sbin/httpd apache 7735 0.1 0.5 211884 11048 ? S 00:36 0:00 /usr/sbin/httpd apache 7761 0.1 0.5 210860 10552 ? S 00:36 0:00 /usr/sbin/httpd apache 7776 0.1 0.4 209836 8648 ? R 00:37 0:00 /usr/sbin/httpd apache 7790 0.2 0.3 208812 7724 ? R 00:40 0:00 /usr/sbin/httpd apache 7800 0.2 0.3 208812 8088 ? R 00:40 0:00 /usr/sbin/httpd root 7801 0.0 0.0 3792 484 tty1 Ss+ 00:41 0:00 /sbin/mingetty tty1 apache 7820 0.2 0.3 208812 7552 ? R 00:41 0:00 /usr/sbin/httpd apache 7834 0.2 0.3 207788 6756 ? R 00:42 0:00 /usr/sbin/httpd apache 7864 0.2 0.2 207788 6148 ? R 00:42 0:00 /usr/sbin/httpd apache 7872 0.3 0.2 207788 5856 ? R 00:43 0:00 /usr/sbin/httpd apache 7874 2.5 0.3 207788 6336 ? R 00:43 0:00 /usr/sbin/httpd root 7875 0.3 0.0 63844 1056 ? S 00:43 0:00 sh -c lsb_release -sd 2>/dev/null root 7879 1.6 0.0 65604 964 pts/0 R+ 00:43 0:00 ps aux root 16316 0.0 0.1 90128 3272 ? Ss Jun05 0:00 sshd: milanb [priv] milanb 16358 0.0 0.0 90128 1752 ? S Jun05 0:00 sshd: milanb@pts/0 milanb 16360 0.0 0.0 66076 1480 pts/0 Ss Jun05 0:00 -bash root 16875 0.0 0.0 101068 1324 pts/0 S Jun05 0:00 su - root 16877 0.0 0.0 66184 1692 pts/0 S Jun05 0:00 -bash root 24373 0.0 0.3 206764 7348 ? Rs Jun05 0:01 /usr/sbin/httpd

UPDATE:

Here, it happens again (about a hour later this time). 

[root@vmi29 ~]# cat /proc/meminfo MemTotal: 2059456 kB MemFree: 17340 kB Buffers: 17788 kB Cached: 407804 kB SwapCached: 0 kB Active: 517704 kB Inactive: 122188 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 2059456 kB LowFree: 17340 kB SwapTotal: 2096472 kB SwapFree: 2096376 kB Dirty: 160 kB Writeback: 0 kB AnonPages: 214436 kB Mapped: 13296 kB Slab: 27392 kB PageTables: 18780 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3126200 kB Committed_AS: 514788 kB VmallocTotal: 34359738367 kB VmallocUsed: 264012 kB VmallocChunk: 34359473911 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB

iostat output:

[root@vmi29 ~]# iostat Linux 2.6.18-194.3.1.el5 (vmi29) 06/06/2010 avg-cpu: %user %nice %system %iowait %steal %idle 6.33 0.00 1.07 1.83 0.00 90.78 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 12.02 44.47 123.40 2526367 7011298 sda1 0.00 0.04 0.10 2036 5466 sda2 0.00 0.03 0.00 1681 200 sda3 12.01 44.39 123.30 2522226 7005632

Free:

[root@vmi29 ~]# free -m total used free shared buffers cached Mem: 2011 2002 8 0 15 386 -/+ buffers/cache: 1600 410 Swap: 2047 0 2047

Command that Nathan Powell suggested:

[root@vmi29 ~]# ps aux | awk '{SUM += $3} END { print SUM }' 20.2

Update2:

[root@vmi29 ~]# ps aux | awk '{SUM += $4} END { print SUM }' 12.5

Free:

[root@vmi29 ~]# free total used free shared buffers cached Mem: 2059456 2038316 21140 0 7360 492800 -/+ buffers/cache: 1538156 521300 Swap: 2096472 100 2096372
Improve this question
13
  • /proc/meminfo should shed some light on what's going on: please post its content. Commented Jun 5, 2010 at 23:49
  • linuxatemyram.com should explain linux memory usage well enough Commented Jun 6, 2010 at 0:11
  • @cpbills: the link you sent in section titled "How do I see how much free ram I really have?" is really the issue I'm talking about here, so I don't see how that helps? "free" shows that I "really have" 400MB free, while "ps" shows that only 700MB is used. 400+700=1100. However, the computer has 2GB of RAM. So, where is 900MB gone? I'm suspecting VMWare issue, but I'd like to be sure before accusing my hosting provider. Commented Jun 6, 2010 at 0:17
  • 1
    I am sorry. My bad. I meant $4 in that command not $3. Commented Jun 6, 2010 at 1:03
  • 2
    The content of meminfo is strange... usually "lost" RAM (not used by the cache) ends up being used by the SLAB, but it's not your case. MemFree (really unsued ram) + Cached + AnonPages + Mapped + Slab should match more or less (there are other allocations, but they're small) the total amount of memory. Are you running a PV kernel? The hypervisor has the ability to forcibly "steal" the memory from guests when need (and of course when it's configured to do it). Commented Jun 9, 2010 at 19:44

4 Answers 4

Reset to default
2

Check vmmemctl memory usage. I had a similar problem, see RedHat Linux: server paging, sum of RES/RSS + buffers + cached < TOTAL. Who is using my memory?

In my case, we had an 8GB RAM server and we couldn't find which process was using. This is our vmmemctl:

cat /proc/vmmemctl target: 1000894 pages current: 1000894 pages rateNoSleepAlloc: 16384 pages/sec rateSleepAlloc: 2048 pages/sec rateFree: 16384 pages/sec timer: 325664 start: 3 ( 0 failed) guestType: 3 ( 0 failed) lock: 3623088 ( 29 failed) unlock: 623698 ( 0 failed) target: 325664 ( 2 failed) primNoSleepAlloc: 3620199 ( 11 failed) primCanSleepAlloc: 2900 ( 0 failed) primFree: 2622165 errAlloc: 28 errFree: 28 getconf PAGESIZE 4096

So vmmemctl is using 4GBs

It's a pity the vmmemctl doesn't use a standard method to report how much memory it's using, but I think it's because how it's implemented.

The main reference from vmware offers a lot of detail about ballooning. I quote since it's relevant to our original problem ( 'why is this server paging if it has non used memory'? ):

"Typically, the hypervisor inflates the virtual machine balloon when it is under memory pressure. By inflating the balloon, a virtual machine consumes less physical memory on the host, but more physical memory inside the guest. As a result, the hypervisor offloads some of its memory overload to the guest operating system while slightly loading the virtual machine. That is, the hypervisor transfers the memory pressure from the host to the virtual machine. Ballooning induces guest memory pressure. In response, the balloon driver allocates and pins guest physical memory. The guest operating system determines if it needs to page out guest physical memory to satisfy the balloon driver’s allocation requests. If the virtual machine has plenty of free guest physical memory, inflating the balloon will induce no paging and will not impact guest performance. In this case, as illustrated in Figure 6, the balloon driver allocates the free guest physical memory from the guest free list. Hence, guest-level paging is not necessary.

However, if the guest is already under memory pressure, the guest operating system decides which guest physical pages to be paged out to the virtual swap device in order to satisfy the balloon driver’s allocation requests. The genius of ballooning is that it allows the guest operating system to intelligently make the hard decision about which pages to be paged out without the hypervisor’s involvement."

"genius of ballooning" :)

Improve this answer
1
  • I wish I could accept 2 answers. troyengel's suggestion helped me for some dedicated servers as well, but in this particular case it was really ballooning effect. The hosting provider oversold the capacity. Commented Oct 9, 2012 at 17:46
2

Run slabtop or parse /proc/slabinfo and look at your kernel slabs; it’s very common for the kernel to cache a lot of directory entries and inodes (dentry_cache, ext3_inode_cache) on a system, especially one with lots of file access like an Apache server that’s sending lots of static content (images, etc.). This is where your “missing” memory is usually.

If this tweaks you out you can adjust vm.cache_pressure in /etc/sysctl.conf to reduce that usage but I highly recommend understanding why first.

Improve this answer
0

VSZ = Virtual Memory Size.

Have a look at all your httpd (apache) processes. 200MB each times lots.

Improve this answer
1
  • After machine is rebooted, I still see 20-30 Apache processes with 200MB VSZ each, but overall RAM usage is only -/+ buffers/cache: 289296. Those 200MB are not multiplied for each Apache process, they are shared. Commented Jun 5, 2010 at 23:35
0

First 'free -m' which shows the usage in Megs, makes more sense visually...since it's not 1996 anymore :)

Second, you have a lot of ram cached, so you aren't out of ram.

3rd, you are only swapping 96K. That is trivial.

To see what is swapping, open 'top', Then use 'O' (cap o) then 'p' then enter.

Ram is not the only thing that will make things "slow". Look at disk i/o and Proc usage as well.

Improve this answer
10
  • Thanks, but it still does not explain why "free" reports 1.6 GB usage while sum of processes is much lower. CPU usage is very high (20+ load) because VMWare's vmmemctl kicks in to deal with the RAM increase. How do I check disk I/O? Commented Jun 5, 2010 at 23:32
  • 1
    @Milan: Use iostat to check I/O Commented Jun 5, 2010 at 23:43
  • Sure it does. Cached is not used. Google "how do I check disk io in Linux" And a load of 20 will certainly be the issue. Commented Jun 5, 2010 at 23:44
  • My question is not why the system is slow. My question is why would "free" report one number, while summing up processes in "ps" output gives a much lower number. Any clues? Commented Jun 5, 2010 at 23:51
  • 3
    If VMware's vmmemctl is active then it is intentionally taking physical RAM away from your VM to give to other VM's. If this is the case then Hypervisor level paging may also be affecting your system. Is it possible that you are seeing a transient side effect of that? What level of vmmemctl ballooning does the Hypervisor report? Commented Jun 6, 2010 at 0:57

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.