Hacker News Reader

Comment by acd | Hacker News Reader

Prometheus.io which is a modern fresh monitoring system that I would checkout if replacing a legacy system.

Also take a look at Riemann which is system monitoring written in Clojure. Riemann should be good for monitoring latency of the system.

If it helps here is Slidedeck from Spotify how they do their monitoring https://www.netways.de/fileadmin/images/Events_Trainings/Eve...

View on HN

I like Riemann and actually read The Art of Monitoring (https://artofmonitoring.com/) which was a great book. There are two big downsides for me on this one. First, you MUST build your monitoring solution from scratch and you MUST learn Clojure (which can be hard to get a whole team to agree to). Second, there's no alerting dashboard, which makes it difficult to see the overall state of the clusters you're dealing with. The only way you know there's a problem if you get an email. Maybe there's a better way to handle that but I wasn't able to find one.

You can alert via a number of mechanisms: email, PagerDuty, Slack et al, etc (I talk about most of those in Chapter 9).

I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console.

Glad you enjoyed the book!

> you MUST learn Clojure (which can be hard to get a whole team to agree to)

For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it.

I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.

It's a major issue for me because of the time investment required to learn the language (functional programming isn't exactly second nature to most programmers) and the need to design the monitoring system from scratch. Instead of an out-of-the-box system you're getting more of a framework which allows you to build one. I actually like Clojure and agree that it is readable as well as fun to learn. But I think with something like Prometheus you do more simple configuration than monitoring system design.

I can only second Prometheus. It's modeled after Google Borgmon (and possibly Monarch) and excels in any flexible workload. Really great piece of engineering.

Riemann is a generic event processor. You can use it to generate alerts or aggregate metrics, but you still need something like collectd or telegraf to collect system/app stats from each machine and send it to Riemann.

Riemann comes with collection clients as well

It should be noted that Riemann is different from Nagios and Prometheus in that it's push-based. The other two are primarily pull-based.

Yep, I think that is Riemann's greatest advantage. I don't like the fact that with Prometheus I have to set up some sort of service discovery mechanism. Right now we're using Foreman to manage our clusters, so most likely I would have to use the file service discovery type and query the Foreman API to produce a YAML or JSON file to provide a list of static hosts.

If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?

Riemann injects expire events into the event stream when it doesn't hear from a service for a while. Each event has a TTL that it uses to figure this out. Then of course you can alert on the expired events or take other actions.

How does this tell you that something is missing if it never reported, or that something is no longer meant to exist?

It's true, you can't tell if something is missing if it never reported. I guess it depends on your requirements. In a situation where your workloads are ephemeral you may not care as much if all of your services have reported as long as most have. In the case that something should no longer exist you could write that functionality into your Riemann configuration. As an example, I could write a "dead service" stream processor that is used to inform Riemann that a service should no longer exist. When a "dead service" event is injected into the event stream for a particular service, Riemann will note the dead service and ignore future events from it.

Reply on news.ycombinator.com