Ask HN: What do you use for Log Management?
In the past few years, there has been a surge in number of log management solutions — Loggly, LogDNA, Scalyr, Sumo Logic. Which one is being used by you / your company?
So let me elaborate. Mostly what will you get from support is: "We are fixing problem", but in our case they were specific, "We have problems with Heroku logspout connection, 'heroku log' should still work." And the other time we went a bit over limit so they upped plan for free for a short period, se we could figure about what the problem was. Alerts are also what we use the most (no limits, no delays) which cannot say for the other providers.
Good work Papertrail, if you are reading this.
Using Graylog at my current job and it's working well so far.
Some of the things I liked about Papertrail:
- Super easy to setup (& automate setup). Just a dozen lines of setup in rsyslog.conf
- Also ships with a tiny client written in go if you want to tail specific files
- Sensible pricing. Charges per GB, which for us at least, correlated with how much business we were doing. Now we'd gladly pay more for the same service
- Great dashboard/UI. Having started with loggly, we were used to slow, unresponsive, unintuitive dashboard. As loggly grew, the dashboard increased in complexity. Papertrail by contrast is fast, simple and made sense (at least to us). It's quite surprising how simple it is and yet performs it's job very well. Although we don't use the live log tailing so much, the log grouping & notification system is very intuitive and easy to setup.
- Easy S3 archival. Took about 10 minutes to setup
Thank you Papertrail
Still getting used to the ways of the cloud, I suppose...
Biggest downside to papertrail is the price. You can log important stuff for a reasonable price. If you want to debug log everything from a fairly big site? You are looking at a papertrail bill 10x your hosting bill.
Not bad idea to send info+ to papertrail, and debug off to S3 or similar directly so you can dig when something weird happens..
- For data exploration, we use memSQL. We keep the last day’s worth of data there(we DELETE rows to keep the memory footprint down), and because most of the time it’s about understanding something that has happened recently, it’s almost always sufficient. Each row contains the event’s representation as JSON, and we also have a few more columns for faster lookup. memSQL’s JSON support is great(we used mySQL for that but it was too slow), so we can take advantage of joins, aggregations, windowing etc.
- For data visualisation, we use ELK (but it’s pretty slow), a tool our ops folks built (“otinanai”: https://github.com/phaistos-networks/otinanai) and we have a few smaller systems that generate graphs and reports.
- For alerts and tickets, our ops folk built another tool that monitors all those events, filters them and executes domain-specific logic that deals with outliers, notifications routing, and more.
This solves most of our needs, but we plan to improve this setup further, by monitoring even more resources and introducing more tools(Tank consumers) to get more out of our data.
[0] https://answers.yahoo.com/question/index?qid=20100103042837A...
Sumo Logic, Graylog, Loggly, PaperTrail, Logentries, Stackify: http://blog.takipi.com/how-to-choose-the-right-log-managemen...
ELK vs Splunk: http://blog.takipi.com/splunk-vs-elk-the-log-management-tool...
Hosted ELK tools: http://blog.takipi.com/hosted-elasticsearch-the-future-of-yo...
We're actually building (and using) a log alternative called OverOps (https://www.overops.com), it's a native JVM agent that adds links to each log warning / error / exception that lead to the actual variable state and code that caused them, across the entire call stack. Disclaimer: I work there, would be happy to answer any question.
Our analysis frontend is plain old SSH, bash, grep and less.
The one I really wanted to use/like was http://scalyr.com. However even after their redesign, I still can't use their query language. With LogEntries, it's pretty natural.
Pros:
* Decent Java logging integration (some services treat things line-by-line, this is a deal breaker for things like multi-line Java exceptions)
* Reasonably priced
* Alerts are kinda nice
* Hosted
Cons:
* Sometimes UI maxes my Chrome CPU
* Live mode not stable at all
* UI is clunky to say the least. It's not always clear what the context of a search is, the autocomplete is obnoxious. I heard they have a new UI coming out sometime, who knows when
I've expanded on this idea here [1]
I am sure the solutions discussed here have features to overcome this (filters / alerts) but IMHO, we'd be better of collecting less things — limited app events that have fixed formatting and are easier to make use of in debugging and monitoring.
You would also get all sorts of benefits, because then you can corelate the app events with user events, and it may be easier to track bugs and unintended behaviour.
Now we use self-hosted ELK (elasticsearch, logstash, & kibana) and I'm not itching to go back to any of the hosted services. It's not as good as something like papertrail for tailing log streams live (although that isn't very useful at larger scale) and the UI of Kibana does take a bit of getting used to though.
We use https://github.com/gliderlabs/logspout to forward all our docker logs to Papertrail... it's like you are watching your nodejs services running in your terminal. Seamless experience.
Disclaimer: I work there :-), happy to answer any of your questions.
I have also have used ELK, but I have to say that SumoLogic felt like a superior product by far.
The stack which we used in our organisation is,
1)Fluentd - for log line transporting 2)Elastic search - for indexing 3)Kibana - for viewing (remote log viewer)
Now a days, on premise, logstash + elastic + kibana.
ELK (logstash, self-hosted) is... consuming. The software is free, but it takes a lot of compute resources, and isn't trivial to come to grips with (setup or daily use). If you can spare the staff-hours, ELK can be pretty powerful, though.