4

I have two machines that are identical in every way (hardware, distribution, application workload, /proc/sys/vm settings, etc), except for the the kernel version. One is 2.6.33 and the other is 3.1.0. The 2.6.33 machine performs significantly better for the given workload (both are I/O bound mostly on reads). I've noticed the Cached/Active/Active(file) is quite a bit smaller on the 3.1.0 machine (over 20GB) and doesn't seem to be accounted for in any other metric being reported. This is also confirmed by the fact that there are a lot more reads happening on the 3.1.0 machine (due to less memory available for page cache most likely). I've looked at every tunable, /proc/buddyinfo for fragmentation, /proc/slabinfo for slab use (uses a a few GB more, but not ~20GB worth), and nothing looks off. Any ideas would be much appreciated.

This is from the machine running 2.6.33 kernel and things look normal.

> cat /proc/meminfo MemTotal: 74372248 kB MemFree: 200492 kB Buffers: 2976 kB Cached: 65324256 kB SwapCached: 0 kB Active: 32949324 kB Inactive: 32689844 kB Active(anon): 287904 kB Inactive(anon): 27272 kB Active(file): 32661420 kB Inactive(file): 32662572 kB Unevictable: 19832 kB Mlocked: 19832 kB SwapTotal: 8393952 kB SwapFree: 8393952 kB Dirty: 8324 kB Writeback: 0 kB AnonPages: 332036 kB Mapped: 12576 kB Shmem: 304 kB Slab: 8217640 kB SReclaimable: 7859644 kB SUnreclaim: 357996 kB KernelStack: 4592 kB PageTables: 10652 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 45580076 kB Committed_AS: 934328 kB VmallocTotal: 34359738367 kB VmallocUsed: 314652 kB VmallocChunk: 34359294955 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 6384 kB DirectMap2M: 2080768 kB DirectMap1G: 73400320 kB

This is from the machine running 3.1.0 kernel. Note the Cached/Active sizes are over 20G smaller and don't seem to be made up in any other metric.

> cat /proc/meminfo MemTotal: 74370628 kB MemFree: 415680 kB Buffers: 384916 kB Cached: 42088392 kB SwapCached: 0 kB Active: 5636160 kB Inactive: 37170092 kB Active(anon): 300656 kB Inactive(anon): 36620 kB Active(file): 5335504 kB Inactive(file): 37133472 kB Unevictable: 19880 kB Mlocked: 7616 kB SwapTotal: 8393956 kB SwapFree: 8393956 kB Dirty: 6524 kB Writeback: 0 kB AnonPages: 354084 kB Mapped: 14588 kB Shmem: 472 kB Slab: 11419580 kB SReclaimable: 9835632 kB SUnreclaim: 1583948 kB KernelStack: 2944 kB PageTables: 12084 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 45579268 kB Committed_AS: 1006028 kB VmallocTotal: 34359738367 kB VmallocUsed: 290448 kB VmallocChunk: 34321698548 kB HardwareCorrupted: 0 kB AnonHugePages: 135168 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 100592 kB DirectMap2M: 6180864 kB DirectMap1G: 69206016 kB
Improve this question
3
  • Are any of these systems being used as hypervisors or are VMs in a hypervisor, or are they real hardware? Commented Mar 11, 2013 at 0:52
  • You may use slabtop which is easier to read than /proc/slabinfo. Sometimes, entries like ext4_inode_cache can be huge if you have millions of files. Also, could you try upgrading the 3.1 kernel to a more recent one (and/or upgrading the 2.6.33 to 3.1) and see if you have the same problem? The fact is you tested it on two different machines, but it would be much better to test different kernels on the same machine, to see if the problem really comes from Linux... Commented Mar 11, 2013 at 1:55
  • Thanks for the suggestions. No hypervisors are being used, just real hardware. I was using slabtop to help examine /proc/slabinfo, along with /proc/pagetypeinfo and /proc/buddyinfo looking for abnormalities. There does appear to be a bit more fragmentation on 3.1, but not enough to account for ~20GB (possibly due to more thrashing from smaller page cache). The 3.1 machine was 2.6.33 but started this slowness and memory difference after the upgrade. I'd like to get to the root of this issue before upgrading the other, but I might try newer versions on the 3.1 machine though. Commented Mar 11, 2013 at 4:11

1 Answer 1

Reset to default
6

It turns out the memory was being consumed by XFS metadata buffers. They moved from the kernel page cache to XFS specific buffers in the 2.6.39 kernel. The patch that changed the behavior can be found here:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0e6e847ffe37436e331c132639f9f872febce82e

The performance difference is due to a different balance between file data and metadata caching as a result of the XFS change.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.