I would like to parse an access log file and have returned the amount of requests for the last 7 days. I have this command
cut -d'"' -f3 /var/log/apache/access.log | cut -d' ' -f2 | sort | uniq -c | sort -rg
Unfortunately, this command returns the amount of requests since the creation of the file and sorts it into HTTP-code categories. I would like just a number, no categories, and only for the last 7 days.
I'd set up log rotation daily (how to do this would be dependent on your OS), then use the same command above on the 7 most recent logs. As for your existing log, either use a tool like grep to extract just the days you want, or split that log into logs for each day.
If you want something more elegant than that, I'd just look for one of the myriad log parsing tools already out there.
Here's an example to split up your existing log: Split access.log file by dates using command line tools
cat day1 day2 day3 day4 day5 day6 day7 > week1, then run your original command on that. It's a Microsoft utility so probably not what you're after but there's a utility called LogParser (link) that will analyse Apache log files and let you use SQL-style syntax to filter, aggregate etc.
You'll want to specify the input format parameter as NCSA.
It should be possible, but I'm getting mired in Bash command nesting that doesn't work, and I don't understand why.
Conceptually, do this:
date -d "-7 days" +%d\/%b\/%Y -> 10/Oct/2013sed '1,/~pattern~/d' access_log| wc -lSo there should be a way to combine the above into one command:
$ sed '1,/10\Oct\/2013/d' access_log | wc -l 29 $ sed '1,/$(date -d "-7 days" +%d\/%b\/%Y)/d' access_log | wc -l $ Somewhere in the nesting, my date command and sed aren't playing nicely. And everything I try with various combinations of quotes and escapes doesn't make any difference.
What am I missing?
How about looking at tools like Splunk or Loggly? Loggly has a free-trial, Splunk Storm (http://splunkstorm.com) is free to sign up, and unless your log files exceed their limits, it should be trivial to get your logs indexed and run various stats on requests in the past 7 days (or various other time frames).
I suggest changing your syslog logfile rotation to be daily then the below script works
#!/bin/bash # I return the sum of hit for gets and posts here. you can remove the final pipe to awk if you want the hits by page ZCAT=$(cat access.log.1 | egrep "GET|POST" | awk '{ print $11 }' ; zcat access.log.[2-7].gz | egrep "GET|POST" | awk '{ print $11 }') echo "$ZCAT" | sort | uniq -c | sort -rg | awk '{ sum=sum+$1 } END { print sum }' This code will allow you to parse out the past 7 days of logs from access.log. it can be used in conjunction with the example in the question or in my other answer to sum the hits for the 7 days parsed out in this example.
#!/usr/bin/bash regex="\*\*\*DONOT_NULLIFY_FORREASONS" days=$(now=`date +%s` for offset in 6 5 4 3 2 1 0 do newdate=`echo "$now - 86400 * $offset" | bc` date +%d/%b/%Y --date="@$newdate" done) for day in $days do regex="$regex"'|'"\[$day" done egrep "$regex" access.log
