1

This server is swapping using 100% (15G swap disk) while it still has 30G free, I've been looking arround but can't figure why this is happening, my understanding was not to use swap if memory is still available. 

 # cat /proc/meminfo MemTotal: 131937984 kB MemFree: 29649644 kB Buffers: 347424 kB Cached: 55013896 kB SwapCached: 2180808 kB Active: 42065900 kB Inactive: 15782332 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 131937984 kB LowFree: 29649644 kB SwapTotal: 16779884 kB SwapFree: 2165300 kB Dirty: 1908 kB Writeback: 0 kB AnonPages: 323104 kB Mapped: 31100628 kB Slab: 408604 kB PageTables: 1737344 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 61777356 kB Committed_AS: 52238240 kB VmallocTotal: 34359738367 kB VmallocUsed: 382524 kB VmallocChunk: 34359355491 kB HugePages_Total: 20480 HugePages_Free: 20480 HugePages_Rsvd: 0 Hugepagesize: 2048 kB # top top - 14:33:16 up 27 days, 21:25, 7 users, load average: 3.47, 4.39, 4.34 Tasks: 669 total, 2 running, 661 sleeping, 0 stopped, 6 zombie Cpu(s): 7.6%us, 1.5%sy, 0.0%ni, 90.2%id, 0.6%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 131937984k total, 101503048k used, 30434936k free, 347544k buffers Swap: 16779884k total, 14608832k used, 2171052k free, 54216540k cached # vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 14608504 30438520 347560 54216984 1 1 503 262 0 0 8 2 90 1 0 3 0 14608504 30439016 347560 54217024 0 0 88 403 1434 4884 0 0 99 0 0 2 0 14608468 30439760 347560 54217060 0 0 10 588 2381 5297 7 1 93 0 0 0 0 14608468 30439884 347560 54217060 0 0 0 76 1429 4768 5 0 94 0 0 1 0 14608448 30440180 347560 54217048 32 0 66 230 3371 4872 4 1 95 0 0 6 0 14608420 30440668 347560 54217076 32 0 42 320 2439 4860 6 1 93 0 0 3 0 14608412 30441384 347560 54217116 0 0 58 520 2128 4899 6 1 93 0 0 4 0 14608412 30441504 347560 54217116 0 0 0 58 1355 4477 11 1 88 0 0 3 0 14608392 30441844 347560 54217012 128 0 158 16 1491 4374 13 1 86 0 0 5 1 14608352 30441512 347560 54216924 160 0 296 640 2748 5279 15 2 83 0 0 4 0 14608324 30442132 347564 54217112 32 0 90 502 2493 4878 13 1 86 0 0 8 0 14608296 30437288 347568 54217204 0 0 8 724 2243 5185 12 1 86 0 0 # cat /proc/sys/vm/swappiness 60 bash-3.2# grep ^[a-z] /etc/sysctl.conf net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.default.log_martians = 0 kernel.panic_on_oops = 1 kernel.panic = 5 fs.aio-max-nr = 399360 kernel.exec-shield = 0 kernel.randomize_va_space = 0 vm.nr_hugepages = 20480 net.ipv4.tcp_keepalive_time 300

An option could be change swappiness from 60 to a lower number

Improve this question

4 Answers 4

Reset to default
1

I have a Sybase server with CentOS 5 which causes no swap. The trick is to set swappiness to 0 and to left enough memory for OS. The server has 16GB (10GB for Sybase and 6GB for OS and other services). Start with a smaller Sybase cache and increase it progressively.

[root@db2 ~]# tail -2 /etc/sysctl.conf # Set Swappiness vm.swappiness = 0
Improve this answer
0

You're probably running software that maintains big caches in process memory. Such pages will be swapped out when another memory-hungry task is run. The memory will not be swapped back until necessary, so when the other memory-hungry task is done, there will be a lot of free memory that won't be filled until necessary.

Improve this answer
5
  • The box is running sybase + VCS, but Sybase was started with 32G memory cap. Is there a way to force the linux not to use the swap if not really needed? Also there is still 29G of mem not in use, I would be happy to track it down, not sure what to do next. Commented Jun 16, 2011 at 13:52
  • Lost of memory seems to go into caching, is that a normal behavior? Commented Jun 16, 2011 at 13:54
  • Seems like you describe normal linux caching behaviour :) Are you sure there has not been a memory intensive task in the past? Commented Jun 16, 2011 at 14:04
  • A database system may be slower when it expects quick access to derived information in a cache. Re-deriving it might be faster. Dropping the caches should cause them to be regenerated in RAM. Commented Jun 16, 2011 at 14:10
  • sar is showing lots of pagin pageout, I need to identify which process is doing it (likely to be sybase) but really wonder why the system is not using the 30G of free memory and goes for the 15G of swap. Commented Jun 16, 2011 at 14:50
0

A friend had this exact problem. I'm assuming you have a multicore system. Being that the case, Linux divides the total physical RAM per processor (ie. if you have a 4core and 16 gigs of ram, it reserves 4gigs per core). When all available RAM is used up for a particular processor, it proceeds to use swap space, instead of taking free RAM from the other processors.

Let me contact him to see what is the exact solution he got, maybe this answer can point to the correct direction in the meanwhile.

Improve this answer
2
  • That would make lot of sense! This server has 32 core and 128G of ram. An other solution would have been to use huge_pages but it's not working with this version of Sybase, that would be great if you frirend has the solution. Commented Jun 16, 2011 at 16:21
  • -1 That answer is wrong on many levels. Commented Feb 24, 2018 at 0:55
0

If you have a single process which has allocated a huge amount of memory on a NUMA system, it may be that it's only letting it have memory belonging to one NUMA node or something.

Have a read of this:

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

The bottom line is: if you can numactl it to use all the memory, it may be better.

However, telling your database to use 32G of 32G seems optimistic, as there will be significant overhead. In particular, on x86_64 the page tables will take up a lot of space (maybe .5 G?) and other parts of the system will need some space. I'd think that about 75% was a better setting.

Improve this answer

You must log in to answer this question.