Bulk imports are planned: https://github.com/prometheus/prometheus/issues/535
Note that bulk really means bulk. This is for one-off loads, not doing push.
708 karma · joined January 3, 2012
Note that bulk really means bulk. This is for one-off loads, not doing push.
For example Prometheus vs InfluxDB is not a great comparison, whereas Prometheus+Alertmanager vs InfluxDB+Kapacitor is.
https://prometheus.io/docs/introduction/comparison/ has such a comparison, though it's a few months out of date.
[Disclaimer: Prometheus developer]
It'll end up about the same in practice, only the time series data that needs to be read is read.
Query performance is looking quite a bit better with this design.
> And I'm really curious how they managed to get 20M writes per second on laptop.
I understand that was a micro-benchmark of one part of the system. The whole system is looking to be roughly in line with the Gorilla numbers.
Consider that a regular scrape has happened and that data has been accepted by the DB but not yet flushed to disk.
Whether the database dies just before or just after the scrape produces the same result: The data for that scrape isn't present when the server restarts.
There plenty of other ways a scrape might not succeed that we have no control over (e.g. other end is overloaded, network blip), so there's not much point obsessing over this particular failure mode.
> Is it that at this ingest rate even trying to ensure durability is a vain effort?
It's not in vain, but it'd be a bad engineering tradeoff in terms of throughput.
The other side of this is that writing out data in that form would naively be an iop per sample, as you're usually appending one sample to 100s of time series in one request.
A significant part of monitoring TSDB design is buffering up samples and batching writes in order reduce that iop rate to something sane.
For example in the right circumstances the 1.x Prometheus design can ingest 250k samples/s on a hard disk which provides ~100 iops/s.
It's presently looking like Influx might once again be an option for long term storage for Prometheus.
We've gone from a shortage of taxis, to an overabundance. When Uber launched, it wasn't interesting as our taxi system (which Uber appears to comply with the laws of) already works.
My experience is that you can get regularly updated runbooks, but they're at the wrong level of abstraction.
They'll discuss some odd one-off failure that happened to trigger a given alert, rather than the general class of problems that this alert is trying to catch.
These days I consider looking at a runbook to be an act of desperation, that'll I'll only perform after attempting to debug from first principles.
I got refunds but nothing beyond that, even with in one case offering evidence that this looked like deliberate fraud due to half-cleared SMART stats.
Indeed. The first order issue is locating the problem though.
If you don't spot which of your microservices is the culprit due to only looking at successful latency, you're not going to get to the stage of comparing successful vs failed latency (and in practice, the increased error ratio combined with increased overall latency should tip you off).
> unless you feed them in to a system that an natively tease them apart as easily as show them together
And the user actually thinks to perform that additional analysis.
> So it seems like the disagreement is more about visualization than collection
What I've seen happen is that the collection leads to the visualisation, which subsequently leads to prolonged outages due to misunderstanding.
Thus I suggest removing the risk of the issue on the visualisation end, by eliminating the problem at the collection stage. This is particularly important when the people doing the visualisation aren't the same people writing the collection code, and thus don't know if the people creating the dashboards will all be sufficiently operationally sophisticated.
> to show "the duration it took to serve a response to a request, also labelled by successes or errors" remains good advice, so long as the visualisation of that data makes clear the separation.
It's a little messier than that. Depending exactly on how the data is collected, such a split could make some analyses more difficult or impossible. For example I need the overall latency increase in order to see if this server is entirely responsible for the overall latency increase I see one level up in the stack, or if there's some other or additional problem that needs explanation. There's no equivalent math for success or failure.
Put another way, the math on the overall works the way your intuition thinks it does. The split out version is more subtle.
>(bias alert - I work on Honeycomb, and care deeply about collecting data in a way that lets you pull away the irrelevant data to illuminate the real problems.)
I work on Prometheus which is a metrics system. Honeycomb seems to be based on event logs. There's logic to removing the success/failure split for duration metrics as I suggest, but it'd be insanity to remove it for event logs. So in your case it is purely a visualisation problem, whereas for us losing granularity at the collection stage is an option (and sometimes required on cardinality grounds).
The terminology the article uses (incrementing a counter at an instrumentation point) led me to believe we were discussing only metrics.
The way I would see things is that you'd use a metrics-based like Prometheus to locate and understand the general problem and which subsystems are involved, and then start using log-based tools like Honeycomb as you dig further in to see which exact requests are at fault. They're complementary tools with different tradeoffs.
I've written about this in more depth at http://thenewstack.io/classes-container-monitoring/
The other way of approaching it is to look for the additional latency it causes, which you can spot on a per-service basis.
I recommend against this, rather have one overall duration metric and another metric tracking a count of failures.
The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed.
The more information you put on a dashboard, the more chance someone will miss a subtlety like this in the interpretation. Particularly if debugging distributed systems isn't their forte, or they've been woken up in the middle of the night by a page.
This guide only covers what I'd consider online serving systems, I'd suggest a look at the Prometheus instrumentation guidelines on what sort of things to monitor for other types of systems: https://prometheus.io/docs/practices/instrumentation/
If you want fast reads, it'd take at least that many IOPS for a naieve solution where each timeseries had its own block you were appending to.
> A typical RDBMS naturally batches parallel writes, so a bunch of 3-4 HDDs providing totally 600+ IOPS would easily serve your situation.
You could do that if you only cared about writes. There's two problems with such an approach.
First reads would unbearably slow as to read a single timeseries for an hour at say 10s resolution would take 360 operations (one per batch), or around 3.6 seconds presuming 10ms seek time. Wanting to read a hundred timeseries at once over a day is not unusual, which would take 8640 seconds.
Secondly it's not likely to be good disk space wise. As each point is individually written you'd not be able to do intra-timeseries compression, so you're probably talking at least 16 bytes per sample. Probably nearer 100-200 bytes, if you write the metric name each time.
Contrast that with the approach taken by something like Prometheus. We build up 1kB chunks, which hold ~780 samples on average as Gorilla gets us ~1.3B/sample. We batch these up per time series across several hours, so accessing 6 hours is only one disk operation. With the above example of reading 100 timeseries for a day that's only 4 seconds, which is 3 orders of magnitude faster than what simple write batching allows for.
Obviously this is ignoring many details like caching, but the general point holds.
Good resolution on the timestamps helps avoid graph artifacts and other weirdness, even if you're only collecting them once a minute. Millisecond accuracy seems to be sufficient in my experience, there's other races of that magnitude anyway (network jitter, kernel scheduling etc.).
One is metrics vs. event logging.
The other is consistency vs. availability.
There's many types of logs, and you don't need the consistency that'd be required for billing-related logs as for debug logs. Similarly you may have billing-related metrics (e.g. bandwidth usage) where consistency is important.
> It's not a _time series_ problem, it's a problem of storage.
Technically anything with a time dimension is a time series, so Splunk is a time series database.
> In my experience the best solution is to use two separate systems for each case.
I agree completely. There's at least two general problems here that need different approaches once you get beyond trivial scale.
For example Prometheus (which I work on) is great at reliable monitoring and powerful processing of metrics at high volumes, but it'd be unwise to use it for event logging or customer billing.
If you're doing IoT or event logging then InfluxDB might be a good choice for you, though if you're doing more text-based logging then Elasticsearch is nearer to what you're looking for.
https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE... is one comparison of the various open source options.
> Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database.
I'm one of the developers of Prometheus, and while I'm going to point appropriate use cases towards Postgres (and regularly do), even relatively small monitoring loads require careful handling such that a traditional database isn't suitable.
An example from previous company is that we had only ~50 machines in one datacenter and yet were up to 30,000 samples per second. This is not considered to be a large setup, which would be in the hundreds of thousands of samples per second.
All the buffering/batching required to make such loads practical requires special design, as naively making each new data point into a disk write (with fdatasync) will not work out - even with SSDs.
And that's just writes. You also need to be able to efficiently read and process back the data efficiently when it is queried.
I'd suggest https://www.youtube.com/watch?v=HbnGSNEjhUc to give a look into how Prometheus does it (which also covers our use of Gorilla). There was also a good post on the InfluxDB blog about the evolution of their design and why the problem is hard that I can't seem find right now.
I've found that giving talks and weekly writing blog posts (https://www.robustperception.io/blog/) are great for acquiring customers. Beware though it's often at least 6-9 months before any of it directly bears fruit.
A few corrections on Prometheus:
> an AlertManager which handles generating and transmitting alerts
Alerts are generated in Prometheus, the Alertmanager then groups, deduplicates and transmits them. See https://www.robustperception.io/prometheus-and-alertmanager-...
> Data Storage Model: Prometheus uses LevelDB as the storage backend
LevelDB is only used for metadata (we used to use LevelDB for everything, that didn't work out too well). The primary timeseries storage is a custom format, which you can find more details about in https://www.youtube.com/watch?v=HbnGSNEjhUc&index=6&list=PLo...
> It has an active community but no enterprise support.
There's several providers of support, my company being one of them. See https://prometheus.io/community/
And some others:
> InfluxDB supports ingesting data via a CLI, client libraries and an HTTP API.
It also accepts a custom TCP protocol, the graphite protocol and the OpenTSDB protocol.
> Influxdb .. supports GroupBy but not TopN queries.
https://docs.influxdata.com/influxdb/v0.13/query_language/fu...
Basically you get 3 things per project where you can use something that's not tried and true.