Prometheus itself has no scalability at all. Without distributed evaluation they have a brick wall.
That's why Google spent all that money to build Monarch. At the end of the day Monarch is vastly cheaper in person time and resources than manually-configured Borgmon/Prometheus. And there is much less friction in trying new queries, etc.
It's not about comparisons, every tool has it's own place and feature set that may be right for you depending on what you're doing. But if you've reached the end of the road with Prometheus due to scale and you need massive scale and perfect compatibility... Then Mimir stands out.
> Federation allows a Prometheus server to scrape selected time series from another Prometheus server
It basically does the opposite of what every scalable system does.
To get HA you double you’re number of pollers.
To get scale your queries you aggregate them into other prometheii.
If this is scalability: everything is scalable.
High Availability always requires duplication of effort. Scaling queries always requires sharding and aggregation at some level.
I've deployed stock Prometheus at global scale, O(100k) targets, with great success. You have to understand and buy into Prometheus' architectural model, of course.
It does not; itself, have highly scalable properties built in.
It does not do sharding, it does not do proxying, it does not do batching, it does not do anything that would allow it to run multiple servers and query over multiple servers.
Look. I’m not saying that it doesn’t work; but when I read about borgmon and Prometheus: I understood the design goal was intentionally not to solve these hard problems, and instead use them as primitive time series systems that can be deployed with a small footprint basically everywhere (and individually queried).
I submit to you, I could also have an influxdb in every server and get the same “scalability”.
Difference being that I can actually run a huge influxdb cluster with a dataset that exceeds the capabilities of a single machine.
https://en.wikipedia.org/wiki/Database_scalability
Scalability means running a single workload across multiple machines.
Prometheus intentionally does not scale this way.
I’m not being mean, it is fact.
It has made engineering design trade offs and one of those means it is not built to scale, this is fine, I’m not here pooping on your baby.
You can build scalable systems on top of things which do not individually scale.
Sorry for being rude, but this level of ignorance is extremely frustrating.
Scalability is defined differently depending on context; in this context (a monitoring/time series solution) it is defined as being able to hold a dataset larger than a single machine that scales horizontally.
Downsampling the data or transforming it does not meet that criteria, since that’s no longer the original data.
The way Prometheus “scales” today is a bolt-on passthrough with federation. It’s not designed for it at all, and means that your query will use other nodes as data sources until it runs out of ram evaluating the query. Or not.
The most common method of “scaling” Prometheus is making a tree; you can do that with anything (so it is not inherent to the technology, thus not a defining characteristic, if everything can be defined the same way then nothing can be- the term ceases to have meaning: https://valyala.medium.com/measuring-vertical-scalability-fo...)
I’ll tell you how influx scales: your data is horizontally sharded across nodes, queries are conducted cross shards.
That’s what scalability of the database layer is.
Not fetching data from other nodes and putting it together yourself.
Rehydrating from many datasets is not the storage system scaling: the collector layer doing the hydration is the thing that is scaling.
If you sold me a solution that used Prometheus underneath but was distributed across all nodes, perhaps we could talk.
But scalability is not a nebulous concept.
You should refer to your own docs if you think Prometheus isn’t a database, it certainly contains one: https://prometheus.io/docs/prometheus/latest/storage/
I should add (and extremely frustratedly): if you’re not lying and you’re a core Prometheus maintainer, you should know this. I’m deeply embarrassed to be telling you this.
This just isn't true :shrug: Horizontal scaling is one of many strategies.
While Prometheus supports sharding queries when a user sets it up, my understanding is that this has to be done manually, which is definitely less convenient. This is better than a hypothetical system that doesn't allow this at all, but still not the same as something that handles scaling magically.
The fact that it could theoretically ingest an infinite amount of data that it cannot thereafter query is not very interesting.
Monitoring as a service has a lot of advantages.
It scales so well that many aggregations are set up and computed for every service across the whole company (CPU, memory usage, error rates, etc.). For basic monitoring you can run a new service in production and go and look at a basic dashboard for it without doing anything else to set up monitoring.
Requiring query authors to understand the arrangement of their aggregation layer seems like a reasonable idea but is in fact quite ridiculous.