27

I'm looking for a way to take a non-intrusive coredump of a running process on Linux.

I'm familiar with gdb's gcore, but that can only be run when gdb is attached to the process and it's stopped for debugging. For a big core dump that might mean many seconds, or even a few minutes, of interrupted execution.

Is there any non-blocking alternative?

Linux supports copy-on-write memory, which it relies upon to support fork() without exec(). So I'm thinking of something kernel-level where the kernel takes a copy-on-write snapshot of the process page tables of the process being dumped, then writes the core out while the original process keeps on running.

I'm pretty sure I could use gdb to force a fork() then dump the child while the parent carries on happily, then wait() in the parent to reap the child after termination. It's messy, though, and still requires two interruptions of the parent process, albeit short ones.

Surely someone's needed this before?

Improve this question
14
  • I am sorry that I can give only a single upvote for this wonderful question. Commented Sep 11, 2014 at 7:36
  • 1
    What about 1) attaching the process with gdb 2) let it be forked by a "call fork" command 3) dumping the core of the child process 4) letting the dead child be waited by the parent (another "call wait4") 5) detach from the process 6) automatize 1-5 ? Gdb uses simple sys_ptrace() system calls, it could be a not really complex C tool totally independent from the gdb. Commented Sep 11, 2014 at 8:09
  • 1
    On a virtual machine you could take a snapshot and bring that up as a clone to be analyze. Perhaps one of the tools listed here will help you: cyberciti.biz/programming/linux-memory-forensics-analysis-tools Commented Sep 12, 2014 at 17:37
  • 1
    You could avoid the second interruption by having the child process also fork and then exit. Then the parent process can wait for the child immediately and then continue, while the grandparent dumps core. Commented Sep 28, 2014 at 0:24
  • 1
    One problem with fork is that it will not duplicate the thread states, you'd need to capture these from the original process. You'd probably anyway need to freeze the process, grab all the thread states, fork to generate the copy-on-write data and then release the original process and save the memory of the new process as the dump. Should be doable but I don't know if an implementatino of this. Commented Oct 14, 2014 at 9:07

2 Answers 2

Reset to default
3

Google CoreDumper springs to mind. It makes a copy-on-write copy of the process's address space, see WriteCoreDump() (see "Notes").

Improve this answer
3
  • That looks exceedingly useful! I wonder what the underlying technique used is. Presumably it ptraces the process, but the creation of the CoW snapshot without forking and in a way that doesn't affect the stack(s) would be challenging. I'll have to take a look at the code. Great tip. Commented Nov 12, 2014 at 4:38
  • Looks like it's in-process only, unfortunately, and can't be invoked via gdb or similar because it requires ptrace its self. So it's a bit like the debughelp DLL under Windows, rather than like a non-blocking gcore, but still very handy looking. I guess it'd be possible to use via an LD_PRELOAD hook and setup of a signal handler with gdb, detach, and signal the process, but it doesn't look like it's really designed to dump unmodified programs, and it has the issue shared by any in-process dump tool that if the process is messed up enough the dump won't work. Commented Nov 12, 2014 at 4:49
  • Sorry… I missed the "non-intrusive" bit when I first read the question. Commented Nov 12, 2014 at 6:49
2

Here's another alternative - Breakpad (mirror on GitHub).

Breakpad is a set of client and server components which implement a crash-reporting system.

Initially it was an in-process only library. But now it contains a pid2md tool, that can generate dumps from a live process, like GDB's gcore, but in minidump format.

The generated minidump file contains enough information (register sets and stacks for each thread) to unwind the thread stacks and display backtraces using the corresponding symbol file. But as far as I understand, it does not contain a snapshot of the heap memory.
Conclusions:

  • The strong point: minidumps are smaller, in my case 10-50x smaller than a classic coredump (even after digging holes in it).
    So, even pid2md works internally like GDB's gcore (stop process and gets memory via ptrace), the time it takes to create a dump should be significantly decreased.
  • The weak point: a post-inspection of the process dump in the debugger is limited.

Note: Breakpad suggests using its own format for symbols. Alternatively, you can convert the generated minidump file into coredump using md2core, and then use it as usual.

Improve this answer
1
  • Thanks very much for the tips. Good to know that breakpad can now be run out-of-process without that process's co-operation. Super useful. Lack of a heap snapshot limits its utility for my purposes though. Commented Apr 9 at 0:01

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.