Hacker News Reader

Comment by freehunter | Hacker News Reader

I'm a big believer in logging, and I find it hard to believe that collecting and monitoring logs needs to have a server with 32GB of RAM and massive Java and ElasticSearch back-ends like ELK, Splunk, Graylog, or similar monitoring software.

I've been searching for something self-hosted that can run in my server's spare capacity and monitor the OS and application logs with a simple searching interface (just a web interface to grep would be handy) and come up short. If I can't host it on the cheapest DO plan, it's out of my budget. I really don't want to have to build it myself but I'm just about at that point.

View on HN

Sounds like you want something like syslog-ng paired with Nagios.

The "big" solutions require some hardware because they do a lot more than merely collecting your logs.

I used to do something similar (but a bit less fancy), with logs processed by Heka + bash scripts using jq. Heka has now been phased out in favor of hindsight[0], but I haven't personally tried that one out yet.

[0]: https://github.com/mozilla-services/hindsight

oklog might be for you? https://github.com/oklog/oklog

It’s still new, but the architecture is sound and it worked well for me on a small single-node deployment (about 30 clients sending logs).

It looks promising. Now if it could integrate easily with syslog-ng/rsyslog, that would be perfect.

Nagios is not-so-easy to configure, but probably will get you most of the way there.

Nagios doesn't do log monitoring out of the box though. Unless I've missed that? It definitely doesn't aggregate logs.

Nagios has check_log but that is not great for memory, I believe it loads the entire log file into memory to take a diff to only check on the entries missed.

There is check_logfile which has a similar name but is a different library, I wrote about how to set this up a while ago: https://medium.com/luma-consulting/how-to-install-check-logf...

It's not super obvious how this plugin works unfortunately. Not the craziest thing in the world to configure but certainly not easy starting from zero!

Reply on news.ycombinator.com