InfluxDB 3.0 System Architecture
influxdata.com
influxdata.com
The switch to a tag based architecture in 1.0 completely broke the database for our use case, it could no longer handle large metric cardinality. Things improved a bit around 1.2, but never got back to something usable for us.
We ultimately moved to using clickhouse for time series data and haven't had to think about it since.
Where is influx at now? Can they handle millions of metrics again? What would bring us back?
(Disclaimer: I work at InfluxData)
InfluxDB 3.0 is optimized for ingestion performance and data compression, paired with a fast columnar query engine. So we can ingest with fewer CPUs, less RAM and reduce storage cost because it's all compressed and put into object store. And we support SQL now (in addition to InfluxQL) with fast analytic queries.
We don't have open source releases yet (that's for later this year), but we have it available in the cloud as a multi-tenant product or dedicated clusters.
And as world seemed to converge on Prometheus/prometheus-compatible interfaces we will probably eventually migrate to VictoriaMetrics or something else "talking prometheus"
InfluxQL was shit. Flux looks far more complex for 90%+ things we use PromQL for now so it is another disadvantage. I'm sure it's cool for data science but all we need to do is to turn some things to rate and do some basic math or stats on it.
> Can they handle millions of metrics again? What would bring us back?
we had one instance with ~25 mil distinct series eating around 26 GB RAM. I'd suggest looking into VictoriaMetrics. Mimir is a bit more complicated to run and seems to require far more hardware for similar performance, but has distinction (whether that's advantage or not, eh...) of using object store instead of plain old disk which makes HA a bit easier.
(Now it’s running in a container, and the container crashes and restarts whenever somebody opens a medium-sized dashboard, causing a few seconds of lost metrics - so being able to OOM-kill the query-daemon while leaving the ingest-daemon running sounds like a step forwards :) )
[1] https://docs.victoriametrics.com/Cluster-VictoriaMetrics.htm...
Personally, if we use InfluxDB in combination with Grafana, the InfluxQL language was quite easy to use in terms of discoverability. With Flux, there's quite a steep learning curve. And mostly clicking something together in the Influx Query builder and copy/pasting it in Grafana, not a fan, although it does work fine.
Flux is an entire language and runtime so we weren't able to implement it natively in Rust. We'll continue to support it for our customers, but the path forward long term is InfluxQL and SQL.
We also submitted a FlightSQL plugin to Grafana that works with InfluxDB 3.0. So either the InfluxDB 1.x plugin using InfluxQL or the FlightSQL plugin work.
So it's planned, but it's going to take a little time.
https://docs.victoriametrics.com/Cluster-VictoriaMetrics.htm...
https://grafana.com/docs/mimir/latest/get-started/about-graf...
Can you shed light on the biggest challenges and steps you (and the team) took to overcome them to succeed with the rewrite? We often hear about how major rewrites can fail or be massively delayed, but you seem to have succeeded.
One of the things that made this tricky is that we weren't just replacing some small system with a single API. We fundamentally changed the underlying architecture of the database and built it around an entirely different paradigm for querying. This is the result of building it around a columnar query engine with a database architecture designed for the cloud and object storage.
So we made a bunch of changes all at once. We didn't start out this way. We wanted to enable some things in the DB like infinite cardinality, tiered data storage, SQL capabilities and a bunch more. When we saw all that, I knew we'd be rewriting the database one way or another.
This was in early 2020. And I figured if we were going to look at some significant rewrite, I'd probably want to do it in Rust. But rewriting your core in a new language is a highly risky endeavor. Honestly, if you can figure out a way to do it iteratively, that's what I'd recommend. A big bang rewrite is the worst possible thing you can do. And it's super stressful.
But... I didn't see a way around that. So we started small with me and one other person working on it starting around March of 2020. Then we added another team member in May (hey Andrew). The three of us spend the next 6 months treating it as a kind of research project. We evaluated building it around existing database engines (like DuckDB and Clickhouse) and looked at what tools we'd want to use.
By August of 2020 we'd settled on building it in Rust with Apache Arrow, Apache DataFusion, and Parquet as the persistence format. I announced this crazy plan in November of 2020 at our online conference and said we were hiring.
Over the first 3 months of 2021 we formed a team around it of 9 people. Everyone else in the company was still focused on everything else we were doing. So the majority of our engineering efforts were focused elsewhere. I think this was critical. Actually, it was quite difficult to have 9 people this early in the project. We hadn't originally planned to scale up that quickly, but we had a flood of great people interested in joining the project (new hires and internal transfers) that we decided to go for it.
Over the next few years we kept this small group working on the new DB while everyone else was working on previous versions of the product. In mid-2022 we were far enough along to bring up the database alongside one of our production environments and start mirroring workloads onto the new DB. This was critical over the following 6 months or so.
We started getting more people from the engineering team looped into the effort in the 4 months leading up to the first launch.
Starting with a small team and scaling up as you get farther along is critical, I think.
There's so much more I could probably write about this, but I'll leave it at this for now :)
InfluxDB v2 was all about the Flux language and a much broader set of API capabilities along with an integrated UI.
For 3.0 we focused on the core database technology. We were able to bring the 1.x APIs forward, which means 3.0 supports both InfluxQL and SQL natively. We were only able to add Flux support through a separate process and a lower level gRPC API that the two use to communicate.
The underlying database architecture is also completely different. v1 and v2 are essentially an inverted index paired with a time series store. v3 organizes data into larger Parquet files and pairs that with a columnar query engine (Apache DataFusion) to execute fast queries against it.
Me or someone on our team should probably write a detailed post about the underlying database architecture to highlight the differences between the versions.
We built 3.0 mainly to accomplish some things that we were unable to deliver in v1 or v2: * Unlimited cardinality * Tiered data storage * Fast analytic queries * SQL compatability * Bulk data import and export (coming soon to v3)
Then there are the systems architecture changes we made highlighted in this blog post. v1 InfluxDB was a monolithic database that had all these components in one. The v3 design allows us to scale ingest, query, and compaction separately, which is something that kept coming up in larger scale use cases.