Buffer/Cache overflow Ubuntu 10.04 LTS Xen Rackspace

Question

I'm running:

Ubuntu 10.04 LTS Server AMD64 in a 512MB XEN paravirtualized environment. lighttpd v1.4.26 PHP v5.3.8 MySQL v5.1.41-3ubuntu12 

The issue is:

I first started on a '256MB account' which is a lie as it registers as 245MB using free -m I found that after I got the system up and running, the memory started slowly disappearing which I later learned was the disk to memory cache/buffer but it still made me nervous as it looked like a memory leak. Sure enough, when it reached the max memory the server crashed. I thought that it was maybe something wrong with my code or that I was using a buggy version of the new PHP. So i changed my cloud size from 256 to 512, which is also a lie, as it reports as 496MB and let that run. After a week, the memory filled up again and the server crashed.

For reference I am not using drupal or php-nuke or anything pre-bundled or bloated. I tried force upgrading from the maximum PHP v5.3.2 LTS release to 5.3.8 My swappiness is set to the default 60.

here is my memory as it sits now:

 total used free shared buffers cached Mem: 496 187 308 0 32 65 -/+ buffers/cache: 89 406 Swap: 1023 0 1023 

The site stays at the 89MB level almost always, but the buffers and cached keep raising. my workaround is I created a daily cron with echo 3 > /proc/sys/vm/drop_caches. This has worked for 3 weeks now but I'm worried that once the site goes mainstream that this hack method will fail. I am here to ask you guys out there that know much more than I do about this scenario, what should I do next?

I'm totally willing to get any data you need to help diagnose this.

My personal hunch is an incompatibility with XEN and Ubuntu, as my guess is XEN says there is a real 512MB of memory to use when its only 496, or that XEN says there is 16GB of memory to use as its incorrect proportioning of the VM. No idea how to confirm that.

Answer 1

Why would you drop buffers and caches? Free memory is wasted memory - all the modern operating systems, including Linux, do aggressively cache things like directory paths, actively used files and so on in RAM, as accessing RAM is insanely fast compared to hard disk.

If some application suddenly needs RAM, cached RAM will be immediately freed for that application. This is a very fast operation and does not cause notable performance hit.

So in Linux you actually calculate your free RAM with total - (free + buffers + cached).

Answer 2

My hunch was right on! It seems that the pre-built PV image i chose did not properly create a swap partition that Linux could identify and use. So after the file cache filled up all the memory, the system would go to cache and either it acted as if there was 0MB swap or it panicked and froze in sort of a cache/swap loop.

USE fdisk -l

here are the examples of different file systems i researched. (PV stands for Paravirtualized, HVM means Fully Virtualized, in a XEN environment)

Custom/Clean HVM Install via CDROM-ISO:

Disk /dev/sda: 16.1 GB, 16106127360 bytes 255 heads, 63 sectors/track, 1958 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0001d37d Device Boot Start End Blocks Id System /dev/sda1 * 1 1871 15021056 83 Linux /dev/sda2 1871 1958 704513 5 Extended /dev/sda5 1871 1958 704512 82 Linux swap / Solaris 

Pre built PV/HVM:

Disk /dev/sda: 16.1 GB, 16106127360 bytes 255 heads, 63 sectors/track, 1958 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00084eb7 Device Boot Start End Blocks Id System /dev/sda1 * 1 13 96256 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 13 75 499712 82 Linux swap / Solaris Partition 2 does not end on cylinder boundary. /dev/sda3 75 1958 15130643 83 Linux 

Failed PV:

Disk /dev/sda1: 20.4 GB, 20401094656 bytes 255 heads, 63 sectors/track, 2480 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sda1 doesn't contain a valid partition table Disk /dev/sda2: 1073 MB, 1073741824 bytes 255 heads, 63 sectors/track, 130 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sda2 doesn't contain a valid partition table