How can I identify which processes are generating UDP traffic on Linux?

Question

My machine is continously making udp dns traffic request. what i need to know is the PID of the process generating this traffic.

The normal way in TCP connection is to use netstat/lsof and get the process associated at the pid.

Is UDP the connection is stateles, so, when i call netastat/lsof I can see it only if the UDP socket is opened and it's sending traffic.

I have tried with lsof -i UDP and with nestat -anpue, but I can't find wich process is doing that request because i need to call lsof/netstat exactly when the udp traffic is sended, if i call lsof/netstat before/after the udp datagram is sended is impossible to view the opened UDP socket.

Call netstat/lsof exactly when 3/4 udp packet is sended is IMPOSSIBLE.

How I can identify the infamous process? I have already inspected the traffic to try to identify the sended PID from the content of the packet, but is not possible to identify it from the contect of the traffic.

Anyone can help me ?

I'm root on this machine FEDORA 12 Linux noise.company.lan 2.6.32.16-141.fc12.x86_64 #1 SMP Wed Jul 7 04:49:59 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

Linux auditing can help. It will at least locate users and processes making datagram network connections. UDP packets are datagrams.

First, install the auditd framework on your platform and ensure that auditctl -l returns something, even if it says that no rules are defined.

Then, add a rule to watch the system call socket() and tag it for easy finding later (-k). I need to assume that you are on a 64-bit architecture, but you can substitute b32 in place of the b64 if you aren't.

auditctl -a exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET 

You have to pick through man pages and header files to build this, but what it captures is essentially this system call: socket(PF_INET, SOCK_DGRAM|X, Y), where the third parameter is unspecified but frequently zero. PF_INET is 2 and SOCK_DGRAM is 2. TCP connections would use SOCK_STREAM which would set a1=1. (SOCK_DGRAM in the second parameter may be ORed with SOCK_NONBLOCK or SOCK_CLOEXEC, hence the &= comparison.) The -k SOCKET is our keyword we want to use when searching audit trails later. It can be anything, but I like to keep it simple.

Let a few moments go by and review the audit trails. Optionally, you could force a couple of packets by pinging a host out on the net, which will cause a DNS lookup to occur, which uses UDP, which should trip our audit alert.

ausearch -i -ts today -k SOCKET 

And output similar to the section below will appear. I'm abbreviating it to highlight the important parts

type=SYSCALL ... arch=x86_64 syscall=socket success=yes exit=1 a0=2 a1=2 ... pid=14510 ... auid=zlagtime uid=zlagtime ... euid=zlagtime ... comm=ping exe=/usr/bin/ping key=SOCKET 

In the above output, we can see that the ping command caused the socket to be opened. I could then run strace -p 14510 on the process, if it was still running. The ppid (parent process ID) is also listed in case it is a script that spawns the problem child a lot.

Now, if you have a lot of UDP traffic, this isn't going to be good enough and you'll have to resort to OProfile or SystemTap, both of which are currently beyond my expertise.

This should help narrow things down in the general case.

When you are done, remove the audit rule by using the same line you used to create it, only substitute -a with -d.

auditctl -d exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET 

Answer 2

You can use netstat, but you need the right flags, and it only works if the process that is sending the data is still alive. It won't find the traces of something that came briefly to life, sent UDP traffic, then went away. It also requires local root privileges. That said:

Here's me starting an ncat on my local host, sending UDP traffic to port 2345 on a (non-existent) machine 10.11.12.13:

[madhatta@risby]$ ncat -u 10.11.12.13 2345 < /dev/urandom 

Here's some tcpdump output proving that the traffic is going:

[root@risby ~]# tcpdump -n -n port 2345 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 12:41:32.391750 IP 192.168.3.11.57550 > 10.11.12.13.2345: UDP, length 8192 12:41:32.399723 IP 192.168.3.11.57550 > 10.11.12.13.2345: UDP, length 8192 12:41:32.401817 IP 192.168.3.11.57550 > 10.11.12.13.2345: UDP, length 8192 12:41:32.407051 IP 192.168.3.11.57550 > 10.11.12.13.2345: UDP, length 8192 12:41:32.413492 IP 192.168.3.11.57550 > 10.11.12.13.2345: UDP, length 8192 12:41:32.417417 IP 192.168.3.11.57550 > 10.11.12.13.2345: UDP, length 8192 

Here's the useful bit, using netstat with the -a flag (to see port details) and the -p flag to see process ID details. It's the -p flag that requires root privileges:

[root@risby ~]# netstat -apn|grep -w 2345 udp 0 0 192.168.3.11:57550 10.11.12.13:2345 ESTABLISHED 9152/ncat 

As you can see, pid 9152 is fingered as having a connection open to port 2345 on the specified remote host. Netstat helpfully also runs that through ps and tells me the process name is ncat.

Hopefully that's of some use.

Answer 3

I had exactly the same problem and unfortunately auditd didn't do much for me.

I had traffic from some of my servers going towards google DNS addresses, 8.8.8.8 and 8.8.4.4. Now, my network admin has mild OCD and he wanted to clean all the unnecessary traffic since we have our intern DNS caches. He wanted to disable outgoing port 53 for everyone except those cache servers.

So, after failing with auditctl, I dig into systemtap. I come up with the following script:

# cat >> udp_detect_domain.stp <<EOF probe udp.sendmsg { if ( dport == 53 && daddr == "8.8.8.8" ) { printf ("PID %5d (%s) sent UDP to %15s 53\n", pid(), execname(), daddr) } } EOF 

Then simply run:

stap -v udp_detect_domain.stp 

This is the output that I got:

PID 3501 (python) sent UDP to 8.8.8.8 53 PID 3501 (python) sent UDP to 8.8.8.8 53 PID 3506 (python) sent UDP to 8.8.8.8 53 

That's it! After changing resolv.conf those PIDs didn't pick up the changes.

Hope this helps :)

Answer 4

Here's a systemtap option, using the netfilter probes available in stap verson 1.8 and later. See also man probe::netfilter.ip.local_out.

# stap -e 'probe netfilter.ip.local_out { if (dport == 53) # or parametrize printf("%s[%d] %s:%d\n", execname(), pid(), daddr, dport) }' ping[24738] 192.168.1.10:53 ping[24738] 192.168.1.10:53 ^C 

Answer 5

I would use a net-sniffer like tcpdump or wireshark to view the DNS requests. The contents of the query can give an idea of what program is issuing them.

Answer 6

Be aware that when using autitctl, nscd for example uses a slightly different parameter in the socket system call, when doing a DNS query:

socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) 

So to make sure you catch those queries in addition to the ones that were mentioned above, you can add an additional filter, with the same name if you want:

auditctl -a exit,always -F arch=b64 -F a0=2 -F a1=2050 -S socket -k SOCKET 

Here 2050 is a bitwise OR of SOCK_DGRAM (2) and SOCK_NONBLOCK (2048).

Then the search will find both those filters with the same key, SOCKET:

ausearch -i -ts today -k SOCKET 

The hex values for the socket constants I found here: https://golang.org/pkg/syscall/#pkg-constants

As I have no reputation points to comment I added this.

Answer 7

I tried the Auditd framework, and initially it didn't look very promising, as all I thought it was telling me was that process sent a UDP packet.

Then tried SystemTap, and initially it worked, but then it started reporting the destination port and address as 0.0.0.0 and 0 on some packets, but not others. Others have had that problem too. Rebooting and clearing caches didn't help, gave up.

Then looked into Dtrace, and quickly transitioned to Perf (thanks, Brendan Gregg!) However, couldn't figure out how to do the same task.

Then found another Auditd solution and apparently Auditd does actually record the destination ports and addresses, but embedded in hex:

$ sudo bash # date +%s; \ auditctl -a exit,always -F arch=b64 -S connect -k sendmsg; \ nslookup serverfault.com 8.8.8.8 > /dev/null; \ auditctl -d exit,always -F arch=b64 -S connect -k sendmsg;\ date +%s 1664829661 1664829661 # grep 1664829661 /var/log/audit/audit.log | grep -C1 -m1 08080808 type=SYSCALL msg=audit(1664829661.285:24390): arch=c000003e syscall=46 success=yes exit=33 a0=14 a1=7f0c3b3f59b0 a2=0 a3=7f0c41415170 items=0 ppid=1523 pid=10483 auid=1003 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2 comm="isc-worker0000" exe="/usr/bin/nslookup" key="sendmsg" type=SOCKADDR msg=audit(1664829661.285:24390): saddr=02000035080808080000000000000000 type=PROCTITLE msg=audit(1664829661.285:24390): proctitle=6E736C6F6F6B7570007365727665726661756C742E636F6D00382E382E382E38 

The saddr=02000035080808080000000000000000 breaks down into:

• 0200 -- PF_INET
• 0035 -- port 53 (DNS)
• 08080808 -- ip address 8.8.8.8

The proctitle decodes as:

# echo 6E736C6F6F6B7570007365727665726661756C742E636F6D00382E382E382E38 | xxd -p -r | xargs -0 nslookup serverfault.com 8.8.8.8 

Also confirmed it with arbitrary destination ip 1.2.3.4:

type=SOCKADDR msg=audit(1664830858.835:24624): saddr=02000035010203040000000000000000 * 0200 -- PF_INET * 0035 -- port 53 (DNS) * 01020304 -- ip address 1.2.3.4 

So... go have fun tracking down processes that send UDP packets! :-)

---

EDIT: Apparently ausearch is a darn useful tool; it converts the arguments to something approximately human-readable:

# ausearch -i -ts today -k sendmsg | more ... type=PROCTITLE msg=audit(2022-10-03 13:58:51.203:24596) : proctitle=nslookup serverfault.com 8.8.4.4 type=SOCKADDR msg=audit(2022-10-03 13:58:51.203:24596) : saddr={ fam=inet laddr=8.8.4.4 lport=53 } type=SYSCALL msg=audit(2022-10-03 13:58:51.203:24596) : arch=x86_64 syscall=sendmsg success=yes exit=33 a0=0x15 a1=0x7f42723ba2e0 a2=0x0 a3=0x7f42723ba520 items=0 ppid=1523 pid=15000 auid=randomuser uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=2 comm=isc-worker0000 exe=/usr/bin/nslookup key=sendmsg ... 

---

EDIT2: Here's live printing of the requests, filtered to just destination port 53 (IDK why destination addr+port is laddr and lport, it just is):

# tail -f /var/log/audit/audit.log \ | ausearch -i -k sendmsg \ | grep -B1 --line-buffered 'lport=53 ' \ | sed 's/ : / :\n /' type=PROCTITLE msg=audit(02/14/2023 16:20:05.022:135695) : proctitle=dig +short www.google.com @8.8.8.8 type=SOCKADDR msg=audit(02/14/2023 16:20:05.022:135695) : saddr={ fam=inet laddr=8.8.8.8 lport=53 } 

---

EDIT3: And here's the TCP version, since DNS can do UDP or TCP:

# auditctl -a exit,always -F arch=b64 -S connect -k MYCONNECT # tail -f /var/log/audit/audit.log \ | ausearch -i -k MYCONNECT \ | grep -B1 --line-buffered 'lport=53 ' \ | sed 's/ : / :\n /' type=PROCTITLE msg=audit(02/14/2023 16:32:18.168:208235) : proctitle=/usr/bin/python3 /home/user/run_test.py type=SOCKADDR msg=audit(02/14/2023 16:32:18.168:208235) : saddr={ fam=inet laddr=127.0.0.1 lport=53 } 

Making it ignore 127.0.0.1 is left as an exercise for the reader. :-)

Answer 8

I have faced the same problem. Could not find any solutions or tools. So I wrote this bash script that co-relates DNS packets (captured with tcpdump) against auditd logs for connect syscall. The co-relation is time based so can have false positives. It has worked well for me so far.

#!/usr/bin/env bash # dns2proc.sh for Linux # Logs DNS queries and correlates them with process information using auditd logs and timing correlation. # Requires: root privileges, auditd, tcpdump # NOTE: Audit rules will be added to the auditd config file and removed on exit # Check for conflicting rules, you may need to remove them manually AUDIT_LOG_FILE="/var/log/audit/audit.log" RED='\033[0;31m' NC='\033[0m' # No Color hex_to_ascii() { # Converts hex string (e.g. 70696E6700676F6F676C652E636F6D) to ASCII echo "$1" | xxd -r -p | tr '\0' ' ' } # Ensure running as root if [[ $EUID -ne 0 ]]; then echo "This script must be run as root." >&2 exit 1 fi # Ensure auditd is running if ! pgrep -x "auditd" > /dev/null; then echo "auditd is not running. Please start auditd." >&2 exit 1 fi # Add audit rules for connect syscall (IPv4 and IPv6) auditctl -a always,exit -F arch=b64 -S connect 2>/dev/null || true auditctl -a always,exit -F arch=b32 -S connect 2>/dev/null || true # Cleanup function to remove audit rules on exit cleanup() { auditctl -d always,exit -F arch=b64 -S connect 2>/dev/null || true auditctl -d always,exit -F arch=b32 -S connect 2>/dev/null || true echo "Cleaned up audit rules." } trap cleanup EXIT echo -e "Listening for DNS queries and correlating with process info using auditd timing..." echo -e "Press Ctrl+C to stop." # More generic regex for tcpdump DNS queries dns_regex='IP ([0-9\.]+)\.([0-9]+) > ([0-9\.]+)\.53: [0-9]+\+[^ ]* ([A-Z]+)\? ([^ ]+)' # Main loop: process each tcpdump line tcpdump -l -nn -i any udp port 53 2>/dev/null | while read -r line; do if [[ "$line" =~ $dns_regex ]]; then SRC_IP="${BASH_REMATCH[1]}" SRC_PORT="${BASH_REMATCH[2]}" DST_IP="${BASH_REMATCH[3]}" DNS_TYPE="${BASH_REMATCH[4]}" DNS_QUERY="${BASH_REMATCH[5]}" # Get the last 3 SOCKADDR events with lport=53 EVENT_IDS=( $(grep 'type=SOCKADDR' "$AUDIT_LOG_FILE" | grep 'lport=53' | tail -3 | grep -oP 'msg=audit\(\K[0-9.]+:[0-9]+') ) if [[ ${#EVENT_IDS[@]} -eq 0 ]]; then echo -e "[$(date)] No audit log found for DNS query: ${RED}$DNS_TYPE $DNS_QUERY${NC} from $SRC_IP:$SRC_PORT" continue fi echo -e "[$(date)] Possible processes for DNS query: ${RED}$DNS_TYPE $DNS_QUERY${NC} from $SRC_IP:$SRC_PORT" declare -A seen_lines for EVENT_ID in "${EVENT_IDS[@]}"; do SYSCALL_LINE=$(grep 'type=SYSCALL' "$AUDIT_LOG_FILE" | grep "$EVENT_ID" | tail -1) PROCTITLE_LINE=$(grep 'type=PROCTITLE' "$AUDIT_LOG_FILE" | grep "$EVENT_ID" | tail -1) if [[ -n "$SYSCALL_LINE" ]]; then COMM=$(echo "$SYSCALL_LINE" | grep -oP 'comm="\K[^ "]+') EXE=$(echo "$SYSCALL_LINE" | grep -oP 'exe="\K[^ "]+') AUDIT_PID=$(echo "$SYSCALL_LINE" | awk '{for(i=1;i<=NF;i++) if($i ~ /^pid=/) {split($i,a,"="); print a[2]}}') AUDIT_PPID=$(echo "$SYSCALL_LINE" | awk '{for(i=1;i<=NF;i++) if($i ~ /^ppid=/) {split($i,a,"="); print a[2]}}') AUDIT_UID=$(echo "$SYSCALL_LINE" | awk '{for(i=1;i<=NF;i++) if($i ~ /^uid=/) {split($i,a,"="); print a[2]}}') AUDIT_EUID=$(echo "$SYSCALL_LINE" | awk '{for(i=1;i<=NF;i++) if($i ~ /^euid=/) {split($i,a,"="); print a[2]}}') AUDIT_AUID=$(echo "$SYSCALL_LINE" | awk '{for(i=1;i<=NF;i++) if($i ~ /^auid=/) {split($i,a,"="); print a[2]}}') USERNAME_UID=$(getent passwd "$AUDIT_UID" | cut -d: -f1) USERNAME_EUID=$(getent passwd "$AUDIT_EUID" | cut -d: -f1) USERNAME_AUID=$(getent passwd "$AUDIT_AUID" | cut -d: -f1) if [[ -n "$PROCTITLE_LINE" ]]; then PROCTITLE_HEX=$(echo "$PROCTITLE_LINE" | grep -oP 'proctitle=\K[0-9a-fA-F]+') CMDLINE=$(hex_to_ascii "$PROCTITLE_HEX") else CMDLINE="(not found)" fi # Lookup parent process info from /proc PARENT_NAME="(not found)" PARENT_EXE="(not found)" if [[ -n "$AUDIT_PPID" && -d "/proc/$AUDIT_PPID" ]]; then if [[ -r "/proc/$AUDIT_PPID/comm" ]]; then PARENT_NAME=$(cat "/proc/$AUDIT_PPID/comm" 2>/dev/null) fi if [[ -L "/proc/$AUDIT_PPID/exe" ]]; then PARENT_EXE=$(readlink -f "/proc/$AUDIT_PPID/exe" 2>/dev/null) fi fi PROC_LINE=" comm=$COMM exe=${RED}$EXE${NC} cmdline=\"$CMDLINE\" PID=$AUDIT_PID PPID=$AUDIT_PPID UID=$AUDIT_UID($USERNAME_UID) EUID=$AUDIT_EUID($USERNAME_EUID) AUID=${RED}$AUDIT_AUID($USERNAME_AUID)${NC} parent_name=\"$PARENT_NAME\" parent_exe=\"${RED}$PARENT_EXE${NC}\"" if [[ -z "${seen_lines[$PROC_LINE]+x}" ]]; then echo -e "$PROC_LINE" seen_lines[$PROC_LINE]=1 fi else echo " No SYSCALL found for event $EVENT_ID" fi done unset seen_lines fi true done 

Source: https://github.com/raj3shp/dns2proc