How can I combine two commands to tally data from compressed and uncompressed log files?

Question

I am running this command to get a tally of failed login attempts from a server’s auth.log and it works well:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 

But the issue is—like every server in the world—I have some GZipped archives created by log rotation that won’t be parsed by this command:

-rw-r----- 1 syslog adm 7822722 Oct 31 13:44 /var/log/auth.log -rw-r----- 1 syslog adm 12532511 Oct 25 06:59 /var/log/auth.log.1 -rw-r----- 1 syslog adm 2250939 Oct 18 06:55 /var/log/auth.log.2.gz -rw-r----- 1 syslog adm 2139669 Oct 11 07:06 /var/log/auth.log.3.gz -rw-r----- 1 syslog adm 2769919 Oct 4 06:54 /var/log/auth.log.4.gz 

To deal with those I have a variant of the command that uses zcat instead of cat but it’s a bit of a nuisance to have to run two commands to get this data. Is there any way to combine the cat and zcat commands into one command so I can get combined results? I know I could write some Bash script to filter through compressed versus uncompressed files, but I use this current one-liner on various servers and I want to have a one-liner I can quickly refer to and use when needed.

More details.

I’ve been doing some security auditing and firewall tweaking and have this command I run that gives me a nice tally of “Failed password” attempts from the auth.log files on an Ubuntu Linux machine:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 

It works great! The output looks something like this:

 5909 Oct 18 13444 Oct 19 351 Oct 20 162 Oct 21 499 Oct 22 377 Oct 23 145 Oct 24 10897 Oct 25 76 Oct 26 54 Oct 27 310 Oct 28 1024 Oct 29 208 Oct 30 30 Oct 31 

And while this works well for uncompressed logs, since the logs rotate and get compressed there are always some GZip compressed files that would be nice to tally up as well. So I run this variant of the above command that uses zcat:

sudo zcat -q /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 

And the output is something like this:

gzip: /var/log/auth.log: not in gzip format gzip: /var/log/auth.log.1: not in gzip format 10413 Sep 27 15977 Sep 28 12297 Sep 29 14438 Sep 30 23394 Oct 1 12912 Oct 2 15844 Oct 3 19697 Oct 4 15350 Oct 5 12358 Oct 6 12692 Oct 7 8377 Oct 8 10875 Oct 9 565 Oct 10 16027 Oct 11 10422 Oct 12 6808 Oct 13 26891 Oct 14 9493 Oct 15 5138 Oct 16 9415 Oct 17 2226 Oct 18 

As you can see, the output works when it works, but it would be nice if the cat and zcat variants of this command could simply be combined into one command. How can that be done?

Bonus Points:

These are not critical issues, but it would be nice if somehow they could be addressed in a solution:

• Note that at the top of the zcat output there are two errors from zcat attempting to process two uncompressed files. Suppressing that would be nice.
• Also note how the cat and zcat commands show data for October 18th; a day when log rotation happened. Would there be any way to add up those two values in the command? If not, I’m fine with having two lines of October 18th data with different values I would have to add up manually.

Answer 1

Can use zgrep which will decompress if necessary and therefore works with both straight text and compressed input. As well grep/zgrep can process multiple files directly which is needed in this case since mixed compressed and text to standard input doesn't always work as intended. Supress filenames in the grep output with -h or --no-filename.

sudo zgrep -h 'Failed password' /var/log/auth.* | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 

Man page:

ZGREP(1) ZGREP(1) NAME zgrep - search possibly compressed files for a regular expression SYNOPSIS zgrep [ grep_options ] [ -e ] pattern filename... DESCRIPTION Zgrep invokes grep on compressed or gzipped files. All options specified are passed directly to grep. If no file is specified, then the standard input is decompressed if necessary and fed to grep. Otherwise the given files are uncompressed if necessary and fed to grep. If the GREP environment variable is set, zgrep uses it as the grep program to be invoked. AUTHOR Charles Levert ([email protected]) SEE ALSO grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1) 

Answer 2

While ugly to type each time, you could do something like:

for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>/dev/null; then sudo cat "$log"; fi; done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 

Probably it would be better to create a function for at least the cat part, something like:

getLogs() { for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>&/dev/null; then sudo cat "$log" fi done } 

Then your command can be

getLogs | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c 

or you can make the whole thing be a function:

getSSHFailures() { for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>/dev/null; then sudo cat "$log" fi done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c } 

Then you just call

getSSHFailures 

You could then define that function in your .bashrc and have that function available in each shell you start

Also, this should also fix both your observation about the error trying to zcat non compressed files (because we redirect zcat errors to /dev/null) and get rid of the duplicate entry, because we're printing all the results together as a single stream before processing them.

Finally, awk can actually do the pattern matching for us, so we can also get rid of the 2 greps like:

getSSHFailures() { for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>/dev/null; then sudo cat "$log" fi done | awk '/sshd.*Failed password/ {print $1,$2}' | sort -k 1,1M -k 2n | uniq -c } 

Answer 3

Do

zcat -f /var/log/auth.log* 

as recommended in answer https://unix.stackexchange.com/questions/77296/is-there-a-tool-that-combines-zcat-and-cat-transparently