Ignore logging for a specific user in Apache httpd

Question

We have setup a system for central logging with the ELK stack (Elastic, Logstash, Kibana). Our shippers (clients) are fluentd that sends massive amount of log data to the ELK. Before fluentd traffic hits logstash it goes through an Apache httpd reverse proxy. It works very well, but we've noticed that the Apache httpd access log is spammed with entries for the fluentd access, like:

10.x.y.z - fluentd [02/Aug/2018:10:31:12 +0200] "POST /elasticsearch/_bulk HTTP/1.1" 200 6471 

We've been testing If blocks to set a variable "dontlogme" and then configured the AccessLog with !dontlogme . This works, if we match for example the User-Agent. But it does not work if we try to make the condition based on (logged in) username.

What we want to achive is: if username (coming from Basic Auth) is "fluentd", dont log the request in access.log .

Is there anyone out there, that have succeeded with this?

Answer 1

If Apache really is incapable of inspecting the "Authorization" header, you can always pipe your access log (as if you were using rotatelogs) through a shell script that filters the fluentd lines.

CustomLog "|/usr/local/bin/nofluentd.sh" combined 

And the nofluentd.sh script something like:

#! /usr/bin/ksh awk '$3 != "fluentd"' | /usr/bin/rotatelogs -n 7 /var/log/apache2/access.log 86400 

Answer 2

As I suspected, Apache has no problem checking the Authorization header with SetEnvIfExpr, after which you can use conditional logging, eg. I created a user fluentd with password haltingd, Apache config as follows:

SetEnvIfExpr "req('Authorization') == 'Basic Zmx1ZW50ZDpoYWx0aW5nZA=='" \ user_fluentd CustomLog "|/usr/bin/rotatelogs -n 7 ${APACHE_LOG_DIR}/druptest.log 86400" \ combined env=!user_fluentd 

Zmx1ZW50ZDpoYWx0aW5nZA== is simply fluentd:haltingd base64 encoded, what a browser will send for basic authentication.