Rsyslog logging to rolled log instead of unrolled log

Question

Forgive me if I am vague, but I am trying to keep this as secure as possible.

I have a CentOS server setup as a central remote logging server. Servers have been sending their logs to this server for a few months now, with no issue. These log files are rolled on a monthly basis, and have been working without incident. This month however, (without any configuration changes that I know of) rsyslog as decided to send new log files to the rolled log file (access_log-20150901), rather then the unrolled file (access_log).

The weird thing is that I even moved the rolled log (access_log-20150901) into an archives directory, and rsyslog is still sending updates to that file.

I am using custom filters, that look like this:

if $programname contains 'access_log' then /var/log/remotelog/access_log 

However it is updating the file at:

/var/log/remotelog/archive/access_log-20150901 

Anyone know what is going on here?

FYI. The permissions are all the same for all of these files and folders.

Edit: Fixed logrotate, with postrotate method to restart rsyslog after the logs have rotated

/var/log/remotelog/access_log { missingok notifempty monthly create 0660 <user> <group> rotate 12 postrotate /sbin/service rsyslog reload > /dev/null 2>/dev/null || true endscript } 

rsyslog version: rsyslogd 8.9.0.ad1

Answer 1

You have two basic options here, IMHO.

1) add a "postrotate" clause to the logrotate stanza to issue a HUP to the rsyslog pid

2) add this file to the rsyslog filespec in the /etc/logrotate.d/rsyslog file.

The problem is that you never restarted the rsyslog daemon after rotating, so it's still holding on to the old filehandle internally. It probably worked fine up until now because the rotate of the rsyslog files took longer than the rotate of this file, which means the HUP on the rsyslog files came after this file had been rotated. Recently, this rotate took longer than the rsyslog files, so the HUP happened before this log file was closed/rotated. After the HUP, rsyslog still had the old file opened, and the rotate never notified rsyslog to update it's internal filehandle.