bbrazil

I'd say we're up to it, and Go is a great language for this class of system. We're however wise enough to know that even with the best people and tools this would take years to get production ready.

> So if your "network falls apart" you have no historical data? I guess thats fine if you're not alerting on data trends, and also have 10 minutes for your graphs to render.

Yes, you'd lose access to historical data (typically more than a few weeks ago). But everything else works, which should include the vast majority of your alerts.

When that came up what you'd do then is refactor that to be a class parameter or config option or whatever (and we'd usually ask that the flag be kept in some form).

Until then though you get the benefits of a quick and easy way to both expose and use tunables, which is much better than not having it at all.

Depends which company. For companies that really care about security, letting arbitrary traffic out is a big no-no. Even via proxy, it requires tight controls.

We've had potential users that were quite excited that Prometheus works the other way, as their network security team were likely to permit it.

> There is a reason why distributed systems exist and that is to scale efficiently with some degree of redundancy baked in.

The efficiency here is in humans (in theory anyway), not in resources or reliability. Distributed systems are a very hard problem, and we avoid those approaches for Prometheus as it's a critical monitoring system. CP systems like Kafka and Zookeeper are not things you want on your alerting path, as they'll fall apart when your network does. Prometheus will keep chugging along.

> Single node HA is probably the most inefficient method of scaling.

I'd disagree, the standard approach these days tends to be a cluster of three which'd use at least 50% more resources.

> It sounds like Prometheus needs to run properly on a single box for simplicity but over time needs to be broken up and made scalable beyond the bounds of a single server.

That's correct. If you manage to have enough targets inside a single datacenter (many thousands of machines), then we recommend vertical sharding first and only if that doesn't work horizontal sharding. Prometheus is really easy to run, so you'll likely end up for organisation reasons choosing to vertically sharding anyway so that each team can control their own decentralised monitoring.

> We didn't want lots of individual consoles.

With Grafana you can view things across many Prometheus servers.

> We wanted a central large system that could hold years of historical data,

Prometheus explicitly doesn't do historical data. As we're effectively talking unbounded amounts of storage that can't fit on one machine, which implies a distributed storage system. As mentioned above that's not something we want in our core for reliability.

The chosen approach is that we'll interface with something else such as OpenTSDB that'll do the long term storage, and we'll support seamlessly graphing across it. That's much easier to make reliable (just add a timeout), and if it does go down you'd still have the last few weeks of data sitting on the Prometheus box.

> as well as allowing people to query it in real time

That we do really well, PromQL is very powerful and anything you can graph you can also alert on.

> and we liked the idea of not needing to double up on hardware costs.

I feel this is a bit of a red herring. The question isn't whether there's a 2X multiplier in the math, it's the overall cost as compared to the benefits.

Prometheus is astoundingly efficient, the latest numbers are 800k samples/s on one machine. I haven't heard of anything else that is even close, and I believe we're also holding the record on storage efficiency.

Even 10X that cost isn't likely to break the bank, so I'd suggest taking a look at the full range of features it offers and comparing to the real world cost. The operational aspects you mention are generally manageable, and if they aren't your infrastructure likely has bigger problems.

Where flags shine is when you've some obscure tunable deep in the dependency tree that you need to tweak (particularly in an emergency). Plumbing through potentially thousands of flags through to main is nonsensical in that scenario.

We do assume that you have your house in order configuration management wise. That's something you may not already have yet at the smaller scales, in which case you may encounter friction as we're not going to add features just to allow you to ignore the need for good configuration management.

For Prometheus at least, we're so efficient that it actually works out okay for the vast majority of users. You'd typically need thousands of instances doing the same thing inside a single datacenter before you get into our (admittedly more involved) horizontal sharding approach.

http://www.robustperception.io/scaling-and-federating-promet... has more information.

> There are ways to poll things and push metrics without opening millions of firewall ports to every security group. Sensu does that quite well, and it scales.

I don't think that's quite a fair comparison. Sensu Just Works when there's no outbound firewall, Prometheus Just Works when there's no inbound firewall. If you add the other direction of firewall for either then things break down.

To make it highly available, just run two identical Prometheus servers. That way if either fails the other is still sending alerts, and if both are working the Alertmanager will automatically deduplicate the alerts.

As for networking, we recommend running Prometheus on the same network as what it's monitoring to avoid crossing failure domains.

You could also open up whatever ports you need on the firewall, just as you would for any other application. Prometheus is no different from any other monitoring solution in requiring firewalls to allow it's traffic through.

Magic Systems is anything other than a manually configured (mostly) simple threshold for alerting.

Bug fixing is still the responsibility of the developers, which isn't to say that SRE won't help out at times but it's not their role.

> If you have legacy codebases run entirely by ops people who don't have any ability to maintain the codebases then you aren't doing SRE.

An SRE is not a maintenance engineer. Service ownership is always a partnership between SRE and developers. If there's no developers, then there's no SREs.

When it comes to alerting, yes. I've seen it tried many times by competent engineers. The problem is that once you get beyond toy examples into situations with even a mere 10k time series there's so much noise that you can't get any useful signal.

> We could really use something like Outalator, though.

I've not found anything like it yet, unfortunately. Hopefully someone will be inspired to write one by the book, there should be enough detail there to do it.

Clustering is really really hard. My personal experience with that class of problem is that it took 3 years to get it production ready, and that was for a simpler problem than what needs to be solved here.

The chances of a 3rd party implementing this given that InfluxData plan to have a commercial version seem very low. I'd expect that most companies who want to stay open source will treat this as a standard database that needs client-side sharding and HA (which is the recommend approach - see https://influxdata.com/high-availability/).

Configuration management is a very different thing to what most of ansible/chef/puppet/salt are commonly used for - it's possible to do it but it's a bit of an uphill battle. The core problem of config management is not pushing out things, but rather how do you represent configs with their myriad of interacting multi-level exceptions while also allowing tens/hundreds/thousands of operational changes to be ongoing at the same time.

Ansible for example is good for about 3-4 levels of exceptions, which isn't that much considering region and environment (dev vs prod) are two of those.

Logs are only one part of monitoring though, and it's easy to miss the wood for the trees if you're drinking from the logs firehose. Some metrics monitoring is also vital, naturally I recommend http://prometheus.io as I'm one of the core developers :) https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ discusses this a bit more.

That's when enterprises might start considering moving onto that version, so the incentives aren't really aligned here.

Basically fire two lasers at right angles to each other at mirrors, and see the pattern when they bounce back.

Getting an Irish bank account is painful though, and that's with me being Irish in Ireland. I wouldn't try it from Portugal - though presumably you could use a local bank.

If there's useful stats from /proc we're missing, we accept PRs.

> about half run on the Prometheus node itself and talk to services like ElasticSearch or Postgres.

That doesn't sound right, those two should run on the ES/Postgres nodes.

Only things like the blackbox exporter and snmp exporter should be considered for running beside Prometheus.

> When I have time, my plan is to write a small HTTP server that spawns plugin as subprocesses that emit their metrics via stdout, which seems like a much more reasonable, low-maintenance solution, and something node_exporter ought to support in the first place.

We went with the textfile collector approach, rather than reinventing cron.

We have many ways to plugin to Prometheus across the ecosystem, the textfile collector you're using is one of them.

> You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ.

There's no fundamental challenge with this approach. If you've got good basic infrastructure, particularly configuration management, the rollout of each should be a small operational task. If it's a major challenge, then your problem probably isn't with the Prometheus architecture.

> it means if we remove a collector, we also have to clean up its .prom files

There's several problems arising from this approach, this is one of them. You can also expect odd artifacts in graphs.

The textfile collector is only intended for machine-level metrics, by putting service level metrics in there you're missing out on a big win of Prometheus by thinking in terms of machines rather than services.

Fighting against the architecture means you're not getting the maximum benefits from Prometheus, this would be easier with exporters and service discovery.

> which themselves need to be monitored

Are you aware that the node exporter exports the mtime of all the textfile collector files? That's there to make monitoring of them easier.

> If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database

As far as we're aware, the only way that happens is if you run out of disk space. If you've evidence otherwise please let us know, so we can prioritize accordingly.

> We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production.

We broke backwards compatibility in the storage format once, and there's no plans to do so again. The core developers who were all running it in production didn't see it as worthwhile to write a converter, and noone else stepped up.

> The alert manager was also written from scratch recently, with a whole new config format.

The old alertmanager has always been flagged as very experimental, as it was a functioning PoC. The rewrite was always been on the cards, and this came up regularly.

This is all part of evolving the system to be better for everyone. If we tried to keep perfect backwards compatibility then we couldn't remove warts, bugs and misfeatures. We aren't afraid to deprecate where it makes sense to do so, and have transition plans where practical.

> The lack of packages (Debian/Ubuntu in our case) is also problematic.

There are packages in Debian proper, and nightlies at http://deb.robustperception.io/

> No syslog support is not acceptable in this day and age.

That's in the latest versions.

The high level problem is that there's so many different ways to do logging that we can't sanely support them all. For every X there is someone who thinks it's essential.

Alerting based on state changes is fragile, it's better to compare against what you expect to see.

If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-m...

This covers not just running the services, but keeping up to date with security updates, scaling the systems, feature additions, bug fixes and keeping up with the state of the art.

Services like RDS are an amazing net cost saving for small to medium users. Something like Cloudfront would cost you a million just to get going. Raw EC2 is nearer break even.

It's the same theory with passports, credit cards etc.

Telegraf can both read and produce Prometheus metrics, which is something I've worked on as I believe metrics shouldn't be locked into any one ecosystem.