708 karma · joined January 3, 2012
There are cases where it'd make sense to live elsewhere though not legitimately, for example the rent-a-room scheme (can rent a room for up to 1k/month tax free) only applies in your PRR. So if you actually lived elsewhere that's a significant fraudulent tax saving if you're a one-property landlord.
The more pertinent difference is event logging vs. metrics.
For the former the ELK stack is popular, and Prometheus.io is suitable for the latter.
Wavefront doesn't publish pricing, but if we take Librato's pricing as a general indication you're talking several million dollars a month.
Prometheus is inspired by Borgmon, and has a query language that is unmatched by almost everything else I'm aware of.
Are there public docs on the semantics and features of the WaveFront language so I can compare?
We skipped the hard stuff on purpose, as hard stuff is extremely tricky to get right and liable to fail right when it's most needed. See http://www.robustperception.io/monitoring-without-consensus/
Per the above, there is HA.
I'd see them as slightly different approaches to providing fundamentally the same solution. One builds up time series and then operates on them, the other operates on the time series as they come in.
Taking Prometheus as an example we're a time series database, and you can do both realtime and window-based analysis. In fact that's how it is usually used.
> I would say that SignalFX is the most sophisticated
Do you have an example of something that you can do with your streaming approach that's not possible with other tools?
It's hard to get a proper understanding of the myriad of monitoring systems out there, so I'm always looking for insights.
> Our agent automatically discovers all the components and dependencies and adds them to the graph in realtime.
That sounds interesting, how do you do that for network dependencies? Do you have something like Zipkin?
How do you re-combine and alert on data for one service that got reported to multiple nodes? This would imply to me a database of some form, which is going to need hashing or federation or other distributed systems approaches to be scaled out.
> Aside from the additional processing requirements
Push or pull, there's additional processing requirements when you scale. It's the same bytes on the wire and broadly the same amount of processing power required.
> you are also limited in that a polling process must be told in advance of new systems or services
That's not a scaling limit, that's a more fundamental issue that isn't different between push and pull.
For a push system you need to have a list of all systems and services in order to be able to alert on systems that never reported, or are no longer reporting.
For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it.
I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.
> [Grafana+Ichinga] they're so rock solid (and scale like mad) I have a hard time not recommending them.
As a Prometheus developer I have seen a significant number of users who moved from Graphite because they found it doesn't scale and was far from rock solid for them, requiring regular manual care and feeding. By contrast Prometheus seems to be working pretty well for them at what we would consider to be a moderate load. I've heard similar about Nagios/Ichinga.
Push vs. pull is largely not relevant for scaling (pull is slightly better in this regard, but only slightly). I've been involved with some extremely large scale monitoring systems, and the fact they were pull was never relevant to scaling them.
May I ask what you consider to be a high level of scale?
> It won't scale without a number of workarounds.
There are very very few systems who won't scale without workarounds. That's the nature of scaling a non-trivial system.
I don't see why processing pipelines would be special for push vs. pull, it's generally a wash.
https://prometheus.io/blog/2016/03/23/interview-with-life360... and https://prometheus.io/blog/2016/05/01/interview-with-showmax... look at two of our users.
You can calculate the highest value now (topk) or the highest averages (topk+avg_over_time) and now that you know which timeseries you want, graph those. I believe this is doable in Grafana.
Ireland has a complex public health system whose cost is comparable to the NHS, it is however not of the same quality.
Systemd is far from everywhere. Ubuntu 14.04 doesn't use it and is supported until 2019. Debian Wheezy is supported until 2018. RHEL 6 has support through to 2021.
Overall this means it'll start to be reasonable to presume systemd sometime around 2020, and be realistically almost everywhere from say 2025.
Software changes take 5-10 years to roll out everywhere. This is such a core change that it'll be on the longer side.
It depends on how many jobs you have, how much churn there is, and what exactly you want to monitor. If jobs are short-lived then tracking individual jobs would be unwise, something like the ELK stack intended for event logging would be better. If jobs are long-lived and there's not many of them then you should be okay. Otherwise you'd just be looking at tracking system rather than per-job stats.
To give a very rough idea, if you can keep it below say 10M metrics across the history a single Prometheus server has that should be okay with the current implementation.
No actually :)
From reading countless websites for the multitude of monitoring solutions out there, the standard pitch is that they'll make your operations more efficient, magically detect problems and give you new insights into your systems.
It's actually really annoying, as you have to dig deep into their docs and squint at screenshots to figure out that actually it's just another Nagios clone :)
> I don't understand. Isn't that twitter blog essentially saying that they went from pull to push? Or is the main point here that they changed to have the agent on each host that collects everything on that host before transferring those metrics to the server (whether that's push or pull)?
What I take from it was had they kept the same design and just changed to push, they'd have continued to have had the exact same problems. I'd view the addition of isolation and health information as what really solved the issue, and whether push or pull is better for that is a wash.
> But I'm not sure Prometheus bends to this either, since IIUC each metric is stored in a separate file, and if we'd have a bunch of metrics for each job ID in the system, this wouldn't really work out, would it?
I'd need a bit more detail to give an exact solution, but Prometheus should work fine for for this.
Metrics are fetched over HTTP by Prometheus, and they usually don't come from files. You'd either have Prometheus scrape each job individually, or scrape your controller daemon that'd attach a labels indicating the job of each metric.
> Did you (or has anyone else) migrate your legacy data from graphite to prometheus?
Prometheus isn't intended as a long-term data store, so there's usually not much point. The data model is also quite different.
> What's the grafana support like on top?
Grafana supports Prometheus as a 1st class integration, and when I was speaking with the Raintank team in person they are very supportive of Prometheus.
> Do you need to run a proxy or something to support the same querying or does it require reworking all the existing graphs?
The data model and query language is more powerful and very different, you have to redo all the graphs.
To aid transition we have https://github.com/prometheus/graphite_exporter which will take in graphite formatted exports from your clients, and convert it to the format Prometheus likes.
> More development resources?
Not directly. I expect the gains to be around the legal and organisational stuff you need as a project grows (e.g. trademark registration, formal governance), as well as publicity.
> I suppose it at least to some extent allays whatever fears there might have been that it will go "open core" in some ways, as e.g. InfluxDB recently..
That was never really a risk, as Prometheus has never been a commercial effort controlled by any one company.
> That being said, what is the killer feature here?
Prometheus is perfect for monitoring any sort of operational timeseries, and is designed to do so in a way that scales both technically and organisationally.
For example collectd is an on-host daemon that you have one of per machine. That may work when you're small, but with many services that will become a bottleneck as each need to be added and stragglers may become an issue (Twitter seem to have hit this, see "Lessons Learned" https://blog.twitter.com/2016/observability-at-twitter-techn...). The Prometheus approach of one exporter per target scales far better.
For another example if two different teams (say infra and database) have two different views of the world, with a collectd approach then the infra team likely wins. With Prometheus monitoring can be decentralised so infra can view a machine as mach1{rack="ch",datacenter="ma"} and database as mach1{owner="app2",env="canary"} each in their own Prometheus server. This gives teams more control by decentralising monitoring.
We're on the look out for suitable remote storage for prometheus.io, and would want to know the hardware that'd be required to handle 1M samples/s and how many bytes a sample takes up.
It doesn't support full float64 which we need, but we could workaround by putting it into a 64 bit unsigned number.
Ignoring that as it's planned work, the typical considerations are more around availability vs. consistency and that we're a metrics system focused on operations rather than a event store. Most of that's already covered in our docs. See https://prometheus.io/docs/introduction/overview/#when-does-... and https://prometheus.io/docs/introduction/faq/
For example if you want to store timeseries data long term and already use HBase then OpenTSDB is a good choice.
On the other hand if you want to do monitoring that's simple and dependable in an emergency with querying, graphing and alerting over short/medium-term data, then Prometheus would a good choice.
To clarify, that's 800k metrics per second per box. That's an upper limit of around 50M metrics with a 60s scrape interval.
It'd be quite difficult to hit this limit with a single service inside one datacenter.
If you do manage to hit it, we do have documented ways to horizontally scale beyond that.