Debugging Apache 2.4 processes

Question

Upgrading from 2.2 to 2.4 somehow mpm_itk was enabled along wiuth mpm_prefork. This problem was discussed in an\my earlier thread : constant mpm_itk errors in Apache error log This resulted in the "mpm_itk errors" going but now I am getting these core notices instead : [core:notice] [pid xxxxx] AH00052: child pid 4597 exit signal Segmentation fault (11) Does anyone know what I should be looking at to finally fix this?

update

I managed to strace a process right through to its segfault, but cant really interpret it. Any help would be greatly appreciated:

rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0 getcwd("/", 4095) = 2 chdir("/website/public") = 0 lstat("/website/public/index.php", {st_mode=S_IFREG|0644, st_size=1777, ...}) = 0 lstat("/website/public", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/website", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={30, 0}}, NULL) = 0 fcntl(14, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f5a3d942028} --- chdir("/etc/apache2") = 0 rt_sigaction(SIGSEGV, {SIG_DFL, [], SA_RESTORER|SA_INTERRUPT, 0x7f5a3cd258d0}, {SIG_DFL, [], SA_RESTORER|SA_RESETHAND, 0x7f5a3cd258d0}, 8) = 0 kill(24177, SIGSEGV) = 0 rt_sigreturn() = 140025556901856 --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_USER, si_pid=24177, si_uid=33} --- +++ killed by SIGSEGV +++ 

Answer 1

The strace command lists the systemcalls the application make as it is running.

If you're not de developer: Section 2 of the system manual documents system calls helping you understand what is happening.

fcntl(14, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 0 

man 2 fcntl

Name fcntl - manipulate file descriptor

int fcntl(int fd, int cmd, ... /* arg */ );

Description
fcntl() performs one of the operations described below on the open file descriptor fd. The operation is determined by cmd.
fcntl() can take an optional third argument. Whether or not this argument is required is determined by cmd. The required argument type is indicated in parentheses after each cmd name (in most cases, the required type is int, and we identify the argument using the name arg), or void is specified if the argument is not required. ...

In other words: a read-lock is attempted on file-descriptor number 14. To find out which file that is there should be symbolic link with the number 14, the integer representing the file in /proc/<PID>/fd/14.

To my eye that looks unrelated to the error in the next line but I'm no developer...:

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f5a3d942028} --- 

si_signo=SIGSEGV looks like the process id your were monitoring received the segmentation fault (SIGSEGV) signal caused by accessing an invalid memory address, 0x7f5a3d942028, an address which it isn't allowed to access (the SEGV_ACCERR si_code).

As to why that happened: strace is often unlikely to help you there.