0

I've run an rsyslog server at our organization for a decade or so now. When we get a new device we want to centralize the logging for, we point it at the rsyslog server's IP address and away it logs.

Every log line has always had a timestamp followed by the sending host's IP address (or sometimes hostname), and until this week I just thought that the rsyslog server added that into the log automatically (possibly doing rdns on it).

But this week we got a new network appliance that sends syslog messages in, and rsyslog records the log lines with no source IP or any other way to distinguish who the sender was, which is not something we've encountered before.

So, I understand I can actually ask the rsyslog server to record the sender IP (for senders who already include their own IP in the message this will create some redundancy but I am fine with that), using templates. But the documentation for how to do templates is complex and difficult to follow, and the rsyslog docs recommend a newer template() syntax anyway which I cannot find many examples for online because everyone still appears to be using $template instead.

I have even tried asking GPT-4 for an example, but it does not appear to be properly versed on the new syntax either and even using the old syntax it adds all kinds of complications into the template that I never asked for and that seriously muddy the waters of properly understanding what syntax I am looking at.

For some extra background: I'm not using any other complexities like forwarders or other software to parse the logs (aside from a handful of scripts I wrote myself that will not be deterred by the planned change in syntax), I just want to be able to grep /var/log/syslog for the sender IP and as a result see what it sent. :)

Improve this question

1 Answer 1

Reset to default
2

Well, after a week of my brain malfunctioning whenever I tried to read these docs, I got a colleague to help me power through them and we came up with this.

The line that saves everything to /var/log/syslog looked like this to begin with:

*.*;auth,authpriv.none -/var/log/syslog

So we created the following template:

template(name="includeLocalTimestampAndSourceIP" type="list") { constant(value="[") property(name="timegenerated" dateFormat="mysql") constant(value=" ") property(name="fromhost-ip") constant(value="]") constant(value=" ") constant(value="(") property(name="timereported" dateFormat="rfc3164") constant(value=" ") property(name="hostname") constant(value=")") constant(value=" ") property(name="syslogtag") constant(value=":") property(name="msg") constant(value="\n") }

and modified the line saving to /var/log/syslog by appending a semicolon and the above template name:

*.*;auth,authpriv.none -/var/log/syslog;includeLocalTimestampAndSourceIP

The property timegenerated is always the time on the rsyslog daemon clock when the packet was received, and setting its format to mysql offers a relatively unambiguous but compact timestamp that looks like 20240227124659. Year, month, day, hour, minute, second with the presumption of local time as perceived by the rsyslog server.

fromhost-ip is the IP address that sent the message, again as perceived by Rsyslog daemon. So even when the host sent a message with a nonsense hostname (like as of this writing all cnMatrix switches sending "ISS" as hostname, or Linksys SIP ATAs that send checks notes the SIP message type as hostname?!) this clarifies who sent this line. Some time down the road I might have Rsyslog save messages from different hosts into different files, but for now I want to stick with what I know works in our workflow.

timereported is the time in the message sent by the host. Sometimes this is off by an hour which means we probably need to fix the host's DST settings, or that the host can't handle DST at all. Sometimes it's off by 8 hours meaning the host is sending UTC.

But most importantly if host reboots and starts sending messages prior to getting an NTP update, reported time will be nonsense (often 1968 Dec 31 as unix timestamp epoch in our time zone, sometimes other arbitrary dates) clarifying which lines were before or after a particular reboot we might be troubleshooting.

Setting its format to rfc3164 creates dates that look like Feb 27 12:46:59, which satisfy those of us on staff who want a general idea what time a log entry was made but don't want to unpack the terse mysql format on one hand, and on the other clarify rapidly to me which timestamp is more trustworthy to be precise vs which one is more informal and thus quite possibly in error.

hostname is what the host wants to be called. For 95% of our gear that is identical to Rsyslog's fromhost-ip, but sometimes a host uses a friendlier name and sometimes errors in this field can be diagnostically relevant. And sometimes it's just garbage lol!

syslogtag is a helpful extra field defined by the host, and scripts I write frequently use this to identify which script is reporting something.

And finally the msg property is the primary payload of the log line.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.