3

We've been having some Linux memory usage problems on some of our servers. free, top, ps, and vmstat are all showing 261M free out of 1G (~25%). However, adding up the amount of memory shown per process in top shows that we should be using very little memory. The cache has only about 48M and 0K in the buffer so that makes the total memory available 309M, still considerably less than we should have. I have included a screen shot of our top output at http://www.workxpress.com/sites/default/files/top.png (sorted by memory). This is my first post on serverfault so I couldn't include the image in my post :-P

After trying several Google searches, I am no where further than I was. The majority of the results told me the same thing, the memory is being used for cache and buffers (which I already knew). But top and free clearly don't show that. Any help is greatly appreciated.

Thanks,
JamesArmes

Update:
I am including our /proc/meminfo from a more recent occurrence of this issue: 

MemTotal: 1028636 kB MemFree: 30056 kB Buffers: 0 kB Cached: 28732 kB SwapCached: 154684 kB Active: 265328 kB Inactive: 19416 kB SwapTotal: 2097144 kB SwapFree: 1758196 kB Dirty: 36 kB Writeback: 0 kB AnonPages: 240260 kB Mapped: 11996 kB Slab: 23008 kB SReclaimable: 11976 kB SUnreclaim: 11032 kB PageTables: 5636 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 2611460 kB Committed_AS: 1099080 kB VmallocTotal: 34359738367 kB VmallocUsed: 7204 kB VmallocChunk: 34359731091 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB

Below is our /etc/sysctl.conf (with all of the commented lines removed):

kernel.printk = 4 4 1 7 kernel.maps_protect = 1 fs.inotify.max_user_watches = 524288 vm.mmap_min_addr = 65536 net.ipv4.conf.default.rp_filter=1 net.ipv4.conf.all.rp_filter=1

I hope this helps.

Improve this question
1
  • 1
    Just so you know, I had exactly this problem with a VMware server, three months ago. We fixed it by turning off the memory limit. I suggest you read up about how VMware handles memory. What is happening is expected. If you get your ESX admin to look at "Memory Balloon Average" in the performance/memory tab of the VCI, it will show where your "missing" memory is. Commented Oct 29, 2009 at 19:33

4 Answers 4

Reset to default
3

Since I see vmware-guestd running, I'll assume that this is a VM. Assuming ESX, go into the VIC and right-click the machine. Click Edit Settings... and select the Resources tab. Click Memory and ensure that Unlimited is check under Limit.

What is likely happening is that you have a limit on the memory, and when the machine uses lots of memory (approaches or exceeds the limit), the vmware guest tools keep the memory "used" rather than releasing it back to ESX, in case it's needed again.

UPDATE:

"The server allows you to power on a virtual machine only if the CPU and memory reservation is available... When resources are not used, the ESX Server host makes them available to other virtual machines."

http://www.vmware.com/pdf/vi3_esx_resource_mgmt.pdf

It does this by way of the "balloon" driver (vmmemctl) which is part of the guest tools. Say your machine is using 100MB of ram, then you run some program and it jumps up to 500MB used. Now you stop that program and expect the RAM to go back down to 100MB. However, it doesn't. This is because, in order for ESXi to reclaim the memory back from the guest OS (which it does even when there is a reservation) it must use the balloon driver. To make the OS "act" as though it has less memory, the balloon driver "uses" the memory that ESXi wants, so that the OS can't use it.

IOW, even with a reservation the guest is only using as much host RAM as it is using. When it uses more, ESX allocates more host RAM which the guest now thinks it always has available. To "convince" the guest that the new ram is gone again, the balloon driver uses it up. The reservation simply means "Don't start the VM unless the physical machine has 1GB free to load the guest into" and not "Give the guest 1GB whether it uses it or not."

Improve this answer
2
  • You are correct in your assumption that we are using ESX (ESXi to be exact). We don't have access to the VIC for this machine, but we know that the limit for this machine is 1G with a reserve of 1G. There is a second machine that is limited to and reserves the majority of what's left on the machine. Commented Oct 29, 2009 at 15:21
  • We took off the limit and it made no difference. However, we did find that the balloon was, on average, taking roughly the same amount of memory as we were missing. Thanks to the document you linked to, we found that we can shut off the ballooning for a specific machine by setting sched.mem.maxmemctl to 0. By doing so, we were able to resolve this issue. While I wouldn't normally recommend this, the ESX server this machine is on is dedicated to two VMs. Commented Oct 31, 2009 at 16:33
1

Try:

cat /proc/meminfo

It gives more details of the memory usage.

Improve this answer
1
  • We tried this, but it didn't lead us anywhere. Commented Oct 29, 2009 at 16:00
1

Try to debug memory usage with the script ps_mem.py

Improve this answer
0

You may have kernel memory reservation. Specifically, check hugepages parameters in /proc/meminfo.

And please post your /etc/sysctl.conf if this suggestion doesn't help.

Improve this answer
4
  • our Hugepagesize is listed as 2048K. I have updated my post to include both /proc/meminfo and /etc/sysctl.conf. Commented Oct 29, 2009 at 17:38
  • Wild guess: reduce max_user_watches. Commented Oct 29, 2009 at 17:59
  • @ynguldyn We don't use inotify so I don't have any idea what a reasonable value for this would be. Do you have any suggestions? Commented Oct 29, 2009 at 18:31
  • Remove the line completely then. Commented Oct 29, 2009 at 19:41

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.