Efficient way to parse Apache Access Log for CMS logins

Question

I've been working on a way to parse large multiple access logs (sometimes in double digit GB in size) for CMSs. Right now I just need WordPress and Joomla, but I know the types will grow over time. Currently I have two oneliners (these will eventually go into a BASH script so any scripting language can be used as long as BASH can interpret it):

echo -e "\n=== WordPress ===\n"; grep --no-messages wp-login.php /usr/local/apache/domlogs/* | grep POST | grep "$(date +"%d/%b/%Y")" | cut --fields=1 --delimiter=: | sort | uniq --count | sort --reverse --numeric-sort | head | sed 's/^ *//g' echo -e "\n=== Joomla ===\n"; grep --no-messages '/administrator/index.php' /usr/local/apache/domlogs/* | grep POST | grep "$(date +"%d/%b/%Y")" | cut --fields=1 --delimiter=: | sort | uniq --count | sort --reverse --numeric-sort | head | sed 's/^ *//g' 

I'm trying to find a way speed these up and combine them so I only need to make one pass and then put them into separate lists. So, for example, the output would look something like this:

=== WordPress === 896 /usr/local/apache/domlogs/blogp.com 52 /usr/local/apache/domlogs/blogt.com 9 /usr/local/apache/domlogs/blog.com === Joomla === 65778 /usr/local/apache/domlogs/bloge.com 478 /usr/local/apache/domlogs/blogq.com 83 /usr/local/apache/domlogs/blogh.com 

I know there are faster parsing methods the just grep are out there, but my knowledge is very limited, so any help would be greatly appreciated.

Answer 1

It depends a bit on how you configure Apache to format your access_log. If you're using the common "CLF" logformat:

LogFormat "%h %l %u %t \"%r\" %>s %b" common CustomLog logs/access_log common 

Then the 7th field contains the request URL. A quick bit of awk which is always present and fairly efficient at parsing text. (Linux almost always has the extended GNU awk as the default; on Solaris you'd better ensure gawk) That allows you to multiple things on a single pass:

awk ' $7 ~ /wp-login.php/ {print $0 } $7 ~ /\/administrator\/index.php/ {print $0} ' access_log 

will search the 7th field $7 with the regular expression ~ enclosed by / forward slashes. When a match is found the {action} enclosed by the braces is performed. A simple action is $print the whole line $0 that contains the match.

A slightly more complex action with updating counters in combination with an END block:

awk ' $7 ~ /wp-login.php/ {++wp } $7 ~ /\/administrator\/index.php/ {++jl} END { print wp , " WordPress Logins found.\n" , jl , " Joomla logins found.\n" } ' access_log 

---

Note: Learn a proper scripting language. That is almost mandatory for a professional system administrator. The actual language is not that important, often a shop has an already established internal scripting standard and then you can have somebody mentor you.
A basic knowledge of bash sed and awk and the assorted commandline utilities in addition to grep will only get you so far. Supplement that by something like perl, python, ruby, even php or PowerShell in a Windows environment.