2

I have very strange case of memory usage on our ubuntu server. One process (which is searchd from sphinxsearch) allocated almost all available memory, and its VSize, RSS and SHR are almost equal (about 18GB). But what makes me really astonished, is that command free treats most of this memory as "cached" - which I always thought is "kernel owned", that is - not bound to particular process. Also, in the same time it's marked as "shared", although there're no other processes with such high memory usage.

So, free -h shows:

root@st3:/proc/31633# free -h total used free shared buffers cached Mem: 23G 22G 649M 18G 62M 18G -/+ buffers/cache: 4.4G 19G Swap: 0B 0B 0B

But for the searchd process we have:

VmPeak: 20451512 kB VmSize: 20413352 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 20325488 kB VmRSS: 20287332 kB VmData: 1344768 kB VmStk: 136 kB VmExe: 4268 kB VmLib: 16204 kB VmPTE: 39924 kB VmSwap: 0 kB

So I can't really understand what is real usage here - it seems most of memory is used only for cache, so it shouldn't be a concern, OTOH we already encountered several failures with "Cannot allocate memory" for simple fork, so that's why I'm trying to understand it.

If you want more, here's the full meminfo from that machine, and here is the searchd process' full list of memory mappings.

And looking at the last one, I see a lot of:

7f7c905b7000-7f7c90713000 rw-s 00000000 00:05 226801755 /dev/zero (deleted) 7f7c90713000-7f7c91fff000 rw-s 00000000 00:05 225400928 /dev/zero (deleted) 7f7c92055000-7f7c921c7000 rw-s 00000000 00:05 226767567 /dev/zero (deleted) 7f7c921da000-7f7c92338000 rw-s 00000000 00:05 226774289 /dev/zero (deleted)

...so I can only guess this could be a point, that searchd is doing some clever tricks to keep memory but in the same point have it available for the system when needed. And maybe it doesn't fully work as expected. But that's only my wild guess, and I can be completely wrong here.

Improve this question

1 Answer 1

Reset to default
1

The 'Cached' entry is counting the number of pages marked as 'shared'. The mappings given are marked as shared.

The kernel internally sees no difference in memory that is set with the shared flag and a file thats simply been catted by the operating system and stored in cache (all files in cache are effectively shared mappings).

The fact this is backed by /dev/zero is inconsequential. The reason they are shared is almost certainly because the same memory data needs to be made available to all child processes that modify the data.

From a semantics perspective it behaves like normally allocated memory (or anonymous memory) given there really is no usable file you can evict pages to (/dev/zero is a not really a file you can save into) and the pages would get moved to swap if they were not in use.

So - confusingly - the data accounts as 'cached' but actually is treated like anonymously backed memory.

You can see this exactly is the case in your meminfo:

root@st3:/proc/31633# cat /proc/meminfo MemTotal: 24690512 kB ... Cached: 19504260 kB ... Active(anon): 3734376 kB Inactive(anon): 18973184 kB Active(file): 227128 kB Inactive(file): 365828 kB ... Mapped: 19237684 kB Shmem: 18985464 kB ...

The same behaviour occurs if you are using IPC shared memory too.

'file' really is whats file backed, 'anon' is whats not got a file backing it -- like your mappings.

Improve this answer
4
  • "The kernel internally sees no difference in memory that is set with the shared flag and a file thats simply been catted by the operating system and stored in cache". But what about re-using this memory? For the (kernel) cached mem, when there's a request for the memory and no free available, cache pages are freed and allocated for the requesting process. But here I guess it's not the case, and this memory is really used - in the sense, that it cannot be used by any other process? So we really lack memory on this machine? Commented Mar 9, 2016 at 14:10
  • As mentioned, its counted as 'cached' (file backed) but treated as anonymous (real memory). Commented Mar 9, 2016 at 14:55
  • So it cannot be reused, and is the reason of our "Cannot allocate memory" problems? Commented Mar 9, 2016 at 15:14
  • Definitely cannot be evicted. Whether its the source of your issues I dont know, although it looks to be using a lot of the memory. Commented Mar 9, 2016 at 18:32

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.