Thoughts on Time-series Databases
jmoiron.net
jmoiron.net
The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along anything but time so every query, for my use case, required a full table scan. 0.9 was supposed to fix every issue I had with it but it was due to be 'production ready' months ago.
Unless you're dealing with ingesting an absolutely insane amount of data indexed along time, I'd have to say that postgres or comparable sql database should be more comfortable, more stable, and much more mature.
EDIT: I don't want to come off as shitting all over influxDB, to its credit it barely moved beyond idle resource usage when I was stuffing it full of data.
Most purpose-built time-series stuff isn't really necessary for much of what I see people trying to use it for.
My question (to everyone): what do you use as replacement for kibana or grafana?
As dashboards for Prometheus, you can currently use PromDash (http://prometheus.io/docs/visualization/promdash/), Console HTML templates (http://prometheus.io/docs/visualization/consoles/), or Grafana (http://prometheus.io/docs/visualization/grafana/).
Durable long-term storage is still outstanding. Although replication into OpenTSDB and InfluxDB is experimentally there.
We've been through a number of data stores ourselves, starting with Postgres back in 2010 - then on to Redis + Cassandra before we built our own. But that's a story for another post...
> 30 seconds
Ugh, that's terrible. grepping the file or reading and parsing a csv file is probably faster.
There's your problem. If you're at the "make it fast" part of "make it work, make it right, make it fast", you should almost certainly be on dedicated hardware.
In 0.8 and before, the preferred way to model your data was to create many separate series names. This method is currently giving many users great performance.
As with all databases, how you model your schema has a significant impact on performance.
That being said, it might be that Influx wasn't right for your use case and Postgres is just a better thing to go with.
Also, we're not supporting anything prior to the 0.9 series of releases. 0.8 is deprecated and we'll be pushing everyone to move over to 0.9 as we put out more point releases and fix bugs and ensure that it works for their use case over the coming months.
Then all the files have a prefix that look like <startsec>_<startmicrosec>_<metric>. A binary search is perfomed first on files in order to pick the right files only. Then the index is read into memory, then binary search performed on the index to get the offset range and then the main data file is read.
When switching to new file time chunk, a cleanup action is peformed where previous old files are removed.
https://influxdb.com/blog/2015/07/02/InfluxDB-0_9_1-and-Tele...
Can you name some examples of this besides the CPU authentication bug? Honest question, I'm considering adding InfluxDB to our stack so I'm genuinely curious.
This by itself wouldn't be a problem for us. But when you're inserting data points, the entire damn thing seems to become unresponsive. Admin UI freezing, http endpoint no longer responding to queries, ...
I know this product is relatively young and your mileage may vary. They have a long way to go.
It pains me to UPDATE and DELETE, thereby destroying useful data.
> to [InfluxDB's] credit it barely moved beyond idle resource usage when I was stuffing it full of data.
solution: always be cramming it full of data ?
I don't see how it could idle when you're bulk writing, but when it's not serving any traffic it's taking up 50% of cpu (unless it's doing some kind of indexing or cleanup or something in the background, but you would expect that to eventually quiesce).
See http://prometheus.io/docs/introduction/faq/#why-does-prometh...
You might think this could be better solved by a simple write-ahead-log (WAL), but the problem is that samples can come in at completely irregular intervals, and we still need to append data from the WAL to the appropriate time series files once chunks for a specific series are full. That would create frequent holes of already persisted data in the WAL, and it would have to be regularly compacted/rewritten. Which is in principle equivalent to the regular checkpointing approach again.
EDIT: There's some more information about Prometheus's storage here: http://prometheus.io/docs/operating/storage/ - but we really should write a white paper or blog post about it soon.
As a simple example, we have one LevelDB index which has single label=value pairs as the keys and as the LevelDB value, the identifiers of the time series which have those label=value dimensions. If you now query for e.g. all time series with labels foo="biz" AND bar="baz", we will do two lookups in that index: one for the key foo="biz", and one for bar="baz". We will now have two sets of time series identifiers which we intersect (AND-style matching) to arrive at the set of time series you're interested in querying. Only then do we actually start loading any actual time series data (not from LevelDB this time).
Baron Schwartz article on Time-series Database Requirements is quite a read: http://www.xaprb.com/blog/2014/06/08/time-series-database-re...
HN discussion: https://news.ycombinator.com/item?id=9166495
It's called multitrack audio recording, and commonly runs on your run of the mill laptop. HD video would probably be a few orders of magnitude more data than that, with even more processing.
Computers are really, really good at these kinds of things if only the software is efficient enough.
Your run of the mill laptop is a wonderful machine capable of amazing things. But that laptop is not rated to withstand extreme vibration and shock and huge temperature ranges. Nor does it have the external interfaces to deal with large numbers of custom, high speed digital data buses, nor does it contain the signal conditioning hardware to deal with a huge array of transducers working with differing physical phenomena, each with its own unique power and signal processing requirements. The system I support has to deal with all of these things, plus more constraints than are relevant to the conversation here.
Much (but not all!) of this doesn't even require a general purpose computer. It can be done with state machines implemented in FPGAs.
Sometimes you need something different than a run of the mill laptop.
Like I said, you do have a very good point. I am often frustrated by the realization that my laptop is much more powerful than this specialized data collection equipment. But we have a very large fixed base of equipment, with no plans to upgrade anytime soon.
EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like something different)
So: some quick googling suggests that "economical cruising speed" of an Airbus A320 is 840km/h [1]. After quick back-of-envelope calculations, this gives ~230m/s, so 0.2m per 0.001s. Given some possible uncertainty of a single measurement, I'd imagine that's not unreasonable level of precision when e.g. your Airbus is landing on an airport.
[1]: http://www.airliners.net/aircraft-data/stats.main?id=23
Part of the reason why it’s hard to learn (unless your job depends on it so you are forced to persist) is that the syntax is very terse. Check out this Java API for example: http://kx.com/q/c/kx/c.java. Yes, that’s the actual code you copy-paste into your Eclipse to get started.
I wrote my own time series database and actually having well working expiry was harder than expected. I sharded each one of the files into 2GB or 24h (whatever came first) chunks. Then had to be careful how they got deleted. There were various rules there such as ("make sure to not fill partition more than 80%", "stop when partition is full" or "keep data no more than 90 days").
My number of different time series was actually pretty low but each value was a record with complex data inside it i.e. not just simple ints or floats. For example, one series was capturing network packets so it looked like [ (timestamp,packet), (timestamp, packet), ... ] . But then an indexing service was running separately in a separate OS process to generate additional complex indexing from the primary data.
Worked surprisingly well, and the indexing overhead was only a few bits per packet (given enough similarity in packets over a short chunk of time (say, 1 minute), then just indexing unique values per chunk -- it was all fairly efficient.)
Main index ( [(timestamp,offset_in_data)] was simply synced every second. It just looked like a [uint64|uint32|...]. Then there is an advanced service with different plugins, runs in a separate process, trails the main data stream. Each plugin can then write out its own custom index data to file and is also saving its internal state (so it can recover during a hard crash).
Fun how designs converge. You'd think there'd be some generic libraries for this kind of stuff. Now there's LevelDB and its SSTable is sort of close but uses Snappy so it can't operate on compressed data.
What's the common library that implements fast, compressed, immutable key values without lots of per record overhead?
All the other products that you mention are for children ;)
I was however focusing on recent Open Source efforts and on the general approaches. Hence, I didn't really discuss in detail my own tsdb.
I also find virtually everything to do with K and Kdb to be be simultaneously impressive and utterly unfathomable:
http://code.kx.com/wsvn/code/kx/kdb%2B/s.k
This is a cheap jab to make, but it makes these systems pretty impenetrable from a source level.
Yes, I know there is a free version, but limited to 32-bit only (and probably non-commercial?).
EDIT: 32-bit version can be used commercially.
As far as readability is concerned, q(KDB+) is far more readable than k(KDB). Also, nobody stops you from adopting a coding style that is more readable. That is what I personally do.
The main downside is HBase is operationally complex, but if you've already made the investment there (as we had), it's a great option.
Cyanite: https://github.com/pyr/cyanite and https://github.com/brutasse/graphite-cyanite
KairosDB: https://github.com/kairosdb/kairosdb and https://github.com/kairosdb/kairos-carbon and https://github.com/Lastik/KairosdbGraphiteFinder
This person has been doing a lot of good work: https://github.com/mwmanley/cyanite
Too many writes, use a cache to batch. No metadata? put the metadata in a different file and point to the offsets.
Why not just use a time decaying ring buffer
It varies but datastores tend to be 1 hour worth of data, then 3 days, then 3 months, then permanent. As you move between datastores latency to access a timestamp increases.
We tend to call Time Series databases "Data Historians" its a big industry and from what I can tell most commercial products are built around ring buffers.
But if you'd actually RTFA, you'd know that Whisper is hardly the end-all of TSD, that cache to batch is not so trivial as you make it seem.
I do believe there's sort of optimal solution that isn't very complex and not far away from many solutions out there. But most definitely more complex than the TSD's in existance today.
You say over-engineered, but if you'd take a look at what's out there, they're all super simple. Usually being a simple layer over some pre-existing storage engine.
I'd be surprised if in any of the currently popular TSD's there more than a month or two of fulltime work put in just the storage engine. Even though there are so many, no one has time to make them as optimized as say popular SQL databases are.
- What about Cassandra/PostgreSQL/Redis?
One of the implications of "99% of data is never read" is that it's incredibly wasteful to keep it all in memory. You might be assuming that you are letting the data expire eventually, but I'm actually not; expiry is a secondary concern to storage.
Once you start to involve the disk (PostgreSQL and Cassandra), you start to get into locality issues that these databases weren't really designed for.
For a more concrete description, lets say I have 2000 machines. Our app is Python, so they run 16 dockerized processes each, with each container reporting 10 simple system metrics every 10 seconds. These metrics are locally aggregated (like statsd) into separate mean, max, min, median, and 90pctile series. I've not instrumented my app yet and that's already 160k writes/sec on average; if our containers all thunder at us it's 1.6m, and frankly this is the "simple" case as:
* we've made some concessions on the resolution
* we only have dense series
Anyone who has used graphite at scale knows that this is a really painful scenario, but these numbers are not particularly big; anywhere you could take an order of magnitude there are a few other places you can add one.
I'm also assuming we are materializing each of these points into their own timeseries, but that's more or less a necessity. It gets back to the locality issues; If we wanted to see the "top 10 system load per node", it's actually quite imperative that we aren't wasting IO cycles on series that _aren't_ system load; we need all we've got to handle the reads.
(As a side point, this is why people in the cloud are adopting so much Go so quickly; it's proven to be easy to write AND read, and also to reduce orders of magnitude in several of the dimensions above, eg. "we can use 1 process per box not 16 containers, and we can get by with 1000 machines not 2000." Having to write your own linked list or whatever doesn't register in the calculus.)
- 1s resolution isn't dense:
No, not always. It's hard to please everyone with these things. In my world, 1s is good (but not great), but 10s seems to more or less be accepted universally, and much sparser (5m, 30m, 6h) is not actually uncommon. At the other end of the spectrum, you can be instrumenting at huge cardinality but very sparsely (think per-user or per-session), perhaps only a few points per timeseries, and the whole of what I've described above kinda gets flipped on its head and a new reality emerges. For what I've described, I quite like the Prometheus approach, but for my very specific use case 1-file-per-metric only beats the filesize block overhead often enough for very long timeframes; to long.
- Why are all TSDB over-engineered?
I hope some of the above has made explicit some of the difficulties in collecting and actually making this data readable. I've only actually thusfar discussed the problem of "associate this {timestamp,value} pair with the correct series in storage"; there are also the following problems:
* you can't query 3 months of 1s resolution data in a reasonable amount of time, so you need to do rollups, but the aggregations we want aren't all associative so you have to do a bunch of them or else you lose accuracy in a huge way (eg, if you do an avg rollup across min data, you flatten out your mins.. which you don't want); this means adding ANOTHER few dimensions to your storage system (time interval, aggregator)
* eventually you have to expire some of this junk, or move it to cold storage; this is a process, and processes require testing, vigilance, monitoring, development, etc.
* you need an actual query system that takes something useful and readable by a human ("AVG system.cpu.usage WHERE env=production AND role=db-master") and determines what series' actually fall into those categories for the time interval you're querying. Anything holistic system that _doesn't_ do this is an evolutionary dead end; eventually, something like Prometheus or Influx will replace them.
These are minimum requirements once you "solve" storage, which is always a very tricky thing to have claimed. If you get here, you've reached what decent SaaS did 4 years ago and what very expensive proprietary systems handled 10 years ago.
- What about Prometheus/InfluxDB/Kdb+/Et al.
Kdb+ is very expensive, its open source documentation is difficult, and its source is unintelligible. It is basically from a different planet that I'm from. Even recently, when I encounter people from, say, the C# world and tell them I work with Python and Go, they ignore Go and say "Wow, there are like no jobs for Python", which I find utterly bewildering. Of course, I never encounter any jobs using C#, either. This is how little some of these spheres overlap sometimes. Someone from the finance world is going to have to come in and reproduce the genius of K and Q for us mortals in a language we understand.
As for Prometheus and InfluxDB, I follow these more closely and have a better understanding of how they operate. I think that they are both doing really valuable work in this space.
From a storage aspect, I think the Prometheus approach is closer to the one that I need for my particular challenges than the InfluxDB one is, and in fact it looks a bit like things we've already had (but also Catena, Parquet, et al..) For most people, storage actually isn't important so long as it's fast enough.
And this is kind of the point of my article. There's starting to be a bit of a convergence among a few Open Source TSDB, and I've taken I've tried to highlight some issues in those approaches and suggest that there's room for improvement. I have my own ideas about what these improvements might look like based on my work at Datadog, and once they're proven (or even disproven) they'll be published somehow.
Those databases are quite raw, to be useful for a use case like monitoring you need to build on top of them to allow for ability to get data from agents, do some processing and enable efficient querying of the data.
http://www.slideshare.net/bijugs/row-or-columnar-based-datab...
then go onto getting mad at Michael Stonebraker here:
https://gigaom.com/2013/12/14/5-predictions-on-the-future-of...
and then when you realize he's right about everything, you'll have the foundational knowledge to understand the problem a little bit better.
If you are at the scale you can't dump it on just 1-2 nodes and call it a day [which is when you are start looking at Cassandra or a dedicated TSD] ...you really need 3 DC availability usually and Redis simply cannot do that in any reasonable way.
* the ingest-side can be scaled up by adding more shards
* the query-side can be scaled up by adding more replicas
From my understanding Postgres has many of these features, but the distributed features of ES are killer!
That said, one major downside to ES is that it's not schemaless. You can try to use the dynamic mapping system, but it will most likely just bite you eventually, since ES is strict about coercing data types. If your data isn't completely consistent, it will actually refuse to index it. Any changes made to your schema also requires reindexing. (For some reason ES can't do in-place indexing, despite supporting storing all the original data in the "_source" field.)
If your data isn't perfectly consistent, one way to work around the mapping problem is to append a type name to every field. So instead of indexing {"user_id": "3"}, you index {"user_id.string": "3"}. This means that if you get some input data where the user_id is an int, it doesn't conflict because it will stored in "user_id.int". You have to handle the inconsistency on the query end, but it's possibly better than micromanaging the index.
If I'm not wrong it combines both very fast in-memory OLAP features and what they call "Deep Storage", which I think is a way to store things on disk for slower historical analysis.
I am leaning towards none of the above being the best solution and am in the process of writing my own (too early to announce yet).
0.8 way okay, but very slow and unstable. But in 0.9 the data model is very different and no longer a good fit for this type of analytics (it looks great for devops metrics, though), so we're abandoning InfluxDB altogether.
We're now in the process of migrating to ElasticSearch, which is looking much better. ElasticSearch has its own problems, though, and we will be evaluating the same dataset in PostgreSQL soon.
Based on that experience, plus from conversations at Monitorma the other week here is what I think of the current state of some various TSDBs are. Some of this might just be lies or rumor - so take it at at that:
* OpenTSDB: Requires HBase behind it, so that can be a pain for people. Maintenance on it is sparse, it doesn't seem like the project has a shortage of contributors with the time needed. Stability isn't great (connection errors from time to time, having alerts based on querying OpenTSDB highlights this). Aggregation and downsampling don't behave as expected. For example rate derivatives happen too late in the order of operations - linear interpolation can be strange. Also to query metric with anything many tag combinations over more than a recent interval of time (say a month or more) is basically impossible - OpenTSDB memory blows up and GC dominates. This requires one to create additional metrics that are denormalized for this. This is kind of okay because OpenTSDB is incredibly storage efficient at storing time series data. No support for NaN. OpenTSDB has quite a bit of serious users https://github.com/OpenTSDB/opentsdb/wiki/Companies-using-Op.... It can ingest a lot of metrics at a high rate without issue.
* KairosDB: Not much experience here. From what I gather it is like OpenTSDB but for Cassandra. Someone mentioned that they thought they heard some core devs have gone to work at Influx which might be concerning - but I don't know if that is true. But same issue of having to run Cassandra if you don't already.
* Graphite: Very rich query language, but currently not a key / value model. Also is not very storage efficient so the approach is that data gets rolled up after a certain period of time - generally problematic for forecasting.
* InfluxDB: Looks promising, but I heard from multiple people "Tried influxdb - was cool but all my data corrupted and I couldn't recover it" at Monitorama. The general concern at Montiroma was that they are overestimating their stability currently when it comes to a production environment. Based on some basic testing at Stack, we found it to be much slower and take up a lot more space than OpenTSDB.
In summary there is no great choice today. More of a pick your pain and best fit situation. But I'm really curious what people with actual experience in these technologies can add to the tradeoffs and am hopeful for the future.
For the 0.9 set of releases, this is what we're supporting going forward. There are some queries that cause the server to crash, but as far as I know, there are no problems that corrupt the database or cause data loss.
We'll be releasing 0.9.1 tomorrow. Every 3 weeks after that we'll be releasing a new point release in the 0.9 line that will be a drop in replacement.
Each one of these releases will fix bugs, improve performance, and add features on clustering (last part starting with 0.9.2).
We're starting work on the on disk size with the 0.9.2 release cycle. If it's ready it'll be in that release in 3 weeks.
Basically, it works now for some use cases and scales. Over the next 3 months we'll be adding features and optimizing to make it useful for larger scales and more use cases.
Overall it's still alpha software, which is why we haven't put anything out there that's called a 1.0 release. However, we're trying very hard to not make any breaking API changes going forward between now and whenever we get to 1.0.
HBase, which is what backs OpenTSDB, can compress data using LZO or snappy. Uncompressed, our test data was at 4GB, but went down to about 400MB after HBase compressed it. InfluxDB was using 8GB. OpenTSDB has done a lot of work to be byte efficient, and it's paid off. We hope InfluxDB will get to a similar place.