5.3.6. Tracking System Call Volume Per Process

⁠5.3.6. Tracking System Call Volume Per Process

This section illustrates how to determine which processes are performing the highest volume of system calls. In previous sections, we've described how to monitor the top system calls used by the system over time (Section 5.3.5, “Tracking Most Frequently Used System Calls”). We've also described how to identify which applications use a specific set of "polling suspect" system calls the most (Section 5.3.4, “Monitoring Polling Applications”). Monitoring the volume of system calls made by each process provides more data in investigating your system for polling processes and other resource hogs.

syscalls_by_proc.stp

#! /usr/bin/env stap# Copyright (C) 2006 IBM Corp.## This file is part of systemtap, and is free software. You can# redistribute it and/or modify it under the terms of the GNU General# Public License (GPL); either version 2, or (at your option) any# later version.## Print the system call count by process name in descending order.#global syscallsprobe begin { print ("Collecting data... Type Ctrl-C to exit and display results\n")}probe nd_syscall.* { syscalls[execname()]++}probe end { printf ("%-10s %-s\n", "#SysCalls", "Process Name") foreach (proc in syscalls-) printf("%-10d %-s\n", syscalls[proc], proc)}

syscalls_by_proc.stp lists the top 20 processes performing the highest number of system calls. It also lists how many system calls each process performed during the time period. Refer to Example 5.18, “topsys.stp Sample Output” for a sample output.

⁠

Example 5.18. topsys.stp Sample Output

Collecting data... Type Ctrl-C to exit and display results#SysCalls Process Name1577 multiload-apple692 synergyc408 pcscd376 mixer_applet2299 gnome-terminal293 Xorg206 scim-panel-gtk95 gnome-power-man90 artsd85 dhcdbd84 scim-bridge78 gnome-screensav66 scim-launcher[...]

To display the process IDs instead of the process names, use the following script instead.

syscalls_by_pid.stp

#! /usr/bin/env stap# Copyright (C) 2006 IBM Corp.## This file is part of systemtap, and is free software. You can# redistribute it and/or modify it under the terms of the GNU General# Public License (GPL); either version 2, or (at your option) any# later version.## Print the system call count by process ID in descending order.#global syscallsprobe begin { print ("Collecting data... Type Ctrl-C to exit and display results\n")}probe nd_syscall.* { syscalls[pid()]++}probe end { printf ("%-10s %-s\n", "#SysCalls", "PID") foreach (pid in syscalls-) printf("%-10d %-d\n", syscalls[pid], pid)}

As indicated in the output, you need to manually exit the script in order to display the results. You can add a timed expiration to either script by simply adding a timer.s() probe; for example, to instruct the script to expire after 5 seconds, add the following probe to the script:

probe timer.s(5){	exit()}