Monitoring Tomcat Instance with Zabbix

Question

Ok, a little back story first,

We have a Tomcat instance that connects to multiple service providers as a sort of "gateway." This Tomcat instance will generate errors if it cannot connect to a given provider. These errors are stored in a log file.

The problem is that we need to monitor the file for more than just a given string, we need to use several regular expressions to filter through all the meaningless stuff so that we aren't waking up at 3am because a user forgot his password. ;)

I already tried creating a python script that would sit between Zabbix and Tomcat, but with log files (that are rotated daily) that are in excess of 200MB, it's too CPU intensive (100% for 15+ seconds). I tried adding a "sleep(0.000005)" to slow it down a bit, but now it takes to long (> 1min) and still uses 25% CPU...

All in all, the solution is overshadowing the problem and I wanted to reach out to you guys and see if you know of a better solution, or anything else that might help.

Answer 1

(1)Zabbix can handle regular expression by itself. You can monitor logs using regex with zabbix agent.

https://www.zabbix.com/documentation/1.8/manual/log_file_monitoring

(2)Using 'nice' command enables you to lower the CPU usage priority of the script.

http://linux.die.net/man/1/nice

Other than that, the more detailed information will be needed. * machine spec (CPU, mem, storage device, etc) * source of the python script

Answer 2

Well, in the end we went with an overcomplicated Python script to do the job until something more viable comes along.

Just to give you an idea of how complicated this "script" is, here are the MINIMUM files/folder that must exist for it to work:

/usr/local/bin/tomcat_log_alert.py (755) /usr/local/bin/tomcat_ack_errors.py (755) /etc/${COMPANY}/tomcat-log-alert.conf (644) /etc/cron.d/tomcat-log-alert (644) /var/log/${COMPANY}/tomcat_errors/ (755) /var/log/${COMPANY}/tomcat_errors_ack/ (755) /etc/zabbix/zabbix_agentd.d/tomcat-log-alert.conf (644) 

I really do not look forward to being the sole maintainer for this fracking band-aid.

Answer 3

The following is worth giving a look:

logrobot autonda /prod/tomcat/error.log 60m 'tomcat errors' 'password forgotten|failed login' 1 2 -show 

The above command will search the log file for any lines that contain "tomcat errors". And from the lines that it finds, it'll remove any lines containing:

• password forgotten
• failed login

The above is only searching ONE log file for the defined entries. What happens if you want to search multiple log files for entries?

You can try something like this:

logrobot automul /prod/tomcat/error.log_MULTIPLE 60m 'tomcat errors' 'password forgotten|failed login' 1 2 -show 

I may be a little off so you should check the either one of these sites to get the exact usage:

• Nagios Log Monitor
• or the home page of the logrobot tool