Hacker News Reader

Comment by falcolas | Hacker News Reader

Just my opinion, but I won't use Prometheus, because of the active polling model. It won't scale without a number of workarounds.

My preferred method is Icinga2 (a Nagios clone with better configuration and clustering built-in) with reports coming in via passive NSCA. Toss in Graphite (or I'm warming up to Grafana on Influx) with some ability to write alerts against those reported metrics, and you're as close to ideal as I can come up with.

Of course, that requires a fair bit of up-front knowledge to stand up and operate, but they're so rock solid (and scale like mad) I have a hard time not recommending them.

View on HN

Active polling scales _better_ than the alternative. You directly manage the ingest rate at the server side, rather than DDOSing your monitoring infrastructure when you scale out your services to deal with load.

> You directly manage the ingest rate at the server side

Why in the world would you want to manage (which I'm reading as throttle) the ingest rate of a monitoring service? That strikes me as a recipe for missing important events.

Monitoring should be a fairly stable load rate. If you're setting up monitoring, and your machine can't handle the load of all the data points coming in, you need to shard out, not drop data points.

> Active polling scales _better_ than the alternative

Exceptional claims require exceptional evidence. Personally, I have never heard of anything that is actively pinging outside services performing better than receiving and processing data passively.

Prometheus will scale better than Nagios running active checks since it won't be using subprocesses, but it is still going to require more overhead than a service receiving passive reports.

The pulling itself has never been a scaling/bottleneck concern in Prometheus, especially since no individual events are transferred, but just the current state of each metric. The bottleneck is usually the storage's sustained ingestion speed (disk IO or similar).

A single big Prometheus server can easily do millions of time series, and in an older record, 800,000 samples stored per second. You could monitor e.g. 10,000 hosts with that with quite some detail, and the bottleneck is still not the pull aspect.

I want to understand more about why you think that pull doesn't scale. Borgmon at Google (the biggest scale ever) worked fine with a pull model, and as far as I know, the successor still pulls, too. A push model also has the problem that now the identity of each instance needs to be known by each instance itself, instead of only by the monitoring system via service discovery (which needs to know about what instances should exist and their identity anyways, otherwise how is it going to let you know that something is wrong?).

We also have an FAQ in Prometheus about why we prefer pull: https://prometheus.io/docs/introduction/faq/#why-do-you-pull...?

In my experience, pulling is operationally much nicer than pushing, and I've worked with both. It also gives you somewhat less of an accidental DDoS exposure.

> but I won't use Prometheus, because of the active polling model.

> [Grafana+Ichinga] they're so rock solid (and scale like mad) I have a hard time not recommending them.

As a Prometheus developer I have seen a significant number of users who moved from Graphite because they found it doesn't scale and was far from rock solid for them, requiring regular manual care and feeding. By contrast Prometheus seems to be working pretty well for them at what we would consider to be a moderate load. I've heard similar about Nagios/Ichinga.

Push vs. pull is largely not relevant for scaling (pull is slightly better in this regard, but only slightly). I've been involved with some extremely large scale monitoring systems, and the fact they were pull was never relevant to scaling them.

May I ask what you consider to be a high level of scale?

> It won't scale without a number of workarounds.

There are very very few systems who won't scale without workarounds. That's the nature of scaling a non-trivial system.

You are wrong about that. There is nothing inherent in a polling model that limits its ability to scale. You simply shard based on service or pod, or row(whatever your unit of scale is) and then you can always use consistent hashing/federation. Prometheus is a single binary. How many moving parts are involved in a Icinga 2 cluster?

Two. Icinga2 and nsca-ng. You can bring it down to one if you use Icinga2 as the test runner on the individual machines.

If you need to shard, no other binaries are involved, just a couple of changed configuration files and some SSL certs. Passive reports can go to any node; a simple load balancer will work fine without any form of hashing or federation required.

> There is nothing inherent in a polling model that limits its ability to scale

Aside from the additional processing requirements, such as SSH, NRPE, subprocesses, etc, you are also limited in that a polling process must be told in advance of new systems or services, whereas it's fairly easy to just have a new service or system start reporting and be immediately monitored.

> Passive reports can go to any node; a simple load balancer will work fine without any form of hashing or federation required.

How do you re-combine and alert on data for one service that got reported to multiple nodes? This would imply to me a database of some form, which is going to need hashing or federation or other distributed systems approaches to be scaled out.

> Aside from the additional processing requirements

Push or pull, there's additional processing requirements when you scale. It's the same bytes on the wire and broadly the same amount of processing power required.

> you are also limited in that a polling process must be told in advance of new systems or services

That's not a scaling limit, that's a more fundamental issue that isn't different between push and pull.

For a push system you need to have a list of all systems and services in order to be able to alert on systems that never reported, or are no longer reporting.

A more thorough investigation of pull vs. push.

https://www.circonus.com/pully-mcpushface/

Not a terribly thorough investigation; simply a series of unsubstantiated assertions.

Pull works fine at 2 million machines" is a great statement to make, but it would be much stronger with more details. How many machines are doing the pulling? How often? Are they using subprocesses or threads or greenthreads? How do they handle timeouts? How many metrics per machine? How many pulls per metric grab?

Reply on news.ycombinator.com