2

I've got a build server where dmesg is reporting that it is having to kill processes because it is running out of memory. As the system is running many builds and other processes concurrently, I need to figure out which process or processes are really using too much memory. i.e. I'm not convinced the process being killed is the one hogging the memory.

Ideally I'd like to dump the memory usage at the point when the out of memory killer kicks in, with a full command line of each process. Is there a way to do this? Alternatively, if I can't dump it at that specific point, I plan to set up a cron job to dump memory usage every minute or two, but I still need some help to get the correct output.

The output from smem is pretty good, but it truncates the command line:

PID User Command Swap USS PSS RSS 39090 user /usr/bin/Xvfb +extension RA 4732 144 148 264 20837 user -bash 0 780 1100 2144 21144 user python /usr/bin/smem 0 12120 12320 13248 19224 user /opt/atlassian/bamboo_home/ 0 234940 235303 237144 12414 user /usr/java/jdk1.8.0_121/bin/ 176128 2249180 2249338 2250428

Is there a way to tell smem to show the full command line? Alternatively a simple way of piping the output to show me what I need? I can pipe into xargs and ps to get the full command line like this:

smem -H -c "pid" | xargs ps

However then I've lost the memory usage values from smem.

Improve this question
2
  • The kernel ring buffer (dmesg) contains the below information about every single process before OOM killer kicks in: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name I believe you're interested in: - total_vm Virtual memory use - rss Resident memory use Commented Feb 9, 2018 at 12:54
  • Thanks, that is useful, but is there a way to get the full command line from dmesg? As I have many processes the same (java + mysql). Commented Feb 9, 2018 at 13:25

3 Answers 3

Reset to default
2

I have simulated a memory hogger with this code:

#include <stdlib.h> #include <stdio.h> #include <string.h> int main(int argc, char** argv) { int max = -1; int mb = 0; char* buffer; if(argc > 1) max = atoi(argv[1]); while((buffer=malloc(1000*1024*1024)) != NULL && mb != max) { memset(buffer, 0, 1000*1024*1024); mb++; printf("Allocated %d MB\n", 1000*mb); sleep(1); } return 0; }

I put the code into the munch.c file and then:

gcc -o munch munch.c ./munch # will run until SIGINT or killed by kernel

I opened two other consoles and run this in them:

# 1st console dmesg -w # 2nd console sudo tail -f /var/log/{kern.log,syslog}

However, I didn't spot anything once then munch process got killed.

So I guess you just need to write a script to monitor the memory consumption. This could be a starting point:

watch 'ps auxw | head -1 && ps auxw | sort -k6 -nr | head -5'

You will get a regulary updated list of processes sorted by the RSS. w option shows full command names.

Improve this answer
1

If you have the process name, you might find something like the answer here quite useful: Finding Average size of single Apache process for setting MaxClients

You can replace httpd at the beginning of that command with your process name and it will show you the total memory usage for processes with that name on the first line and the average memory usage of those processes on the second line. Hope this helps! :)

Improve this answer
0

have you tried using top and using c toggle so you can see the process name and command line?

https://en.wikipedia.org/wiki/Top_(software)

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.