0

I am thinking about building an application that would read my different server log files, save the parsed data to a database, and then remove the line from the log file.

This is my programer side trying to think of better ways to get all the log files into a useable/searchable application console I can view multiple servers at a time from and program scripts to find correlations or email/txt me while I'm away when something looks wrong.

There seem to be expensive solutions for this already on the market, but I can't justify the hefty $500-$2000/mo prices.

Anyway, my problem is how do I remove lines from an actively written log file without causing a problem? Some of the log files are rotated (like nginx) while others are not. I suppose the safest thing would be to copy the file, then echo '' > file.log to erase it. I might miss 300ms of writes though.

The other question is whether I even want to erase the logs. I suppose not erasing the logs means I would have to open the file, jump to the end, then work my way backwards until I got to the last known entry.

What would be a good way to pipe log data to an external application/database?

Improve this question
1
  • 1
    Advice - If you're going to dip your toes into systems administration, you'll definitely want to do research on best-practices and how others have already dealt with the overall problem you're facing (not automatically digging into details like how to deal with open file handles, etc). To rotate logfiles, you can use something like (wait for it) logrotate. As Mr Hampton says below, don't reinvent the wheel. Sysadmins have a lot of wheels. Commented Dec 31, 2014 at 17:41

2 Answers 2

Reset to default
3

Don't reinvent the wheel.

Use logstash to get your logs off your systems.

Have logstash send the logs to elasticsearch.

Use the kibana front end for analytics.

This combination is so common it's known as the ELK stack. And it's all open source.

Improve this answer
1
  • ELK looks perfect! Commented Dec 31, 2014 at 17:17
1

syslog-ng​​​​​​​​​​​​​​​​​​​​​

Improve this answer
2
  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. Commented Dec 31, 2014 at 17:10
  • 1
    @NathanC In my weak defense, twice the question How to pipe all log files entries to a database on linux? was asked, and the answer is one word: syslog-ng. Nothing more needs to be said. It's a perfect and full answer to the question. The wikipedia article was just gravy, so this isn't a link only answer, at least not in spirit. Anything more would be outside the question's scope (like, "halp how to syslog my ngs?!"). And yes, it's a weak defense. =) Commented Dec 31, 2014 at 20:31

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.