pauldix

In this case, the features we kept getting asked for by our customers necessitated a change in underlying database architecture. I talk about that quite a bit in the reddit thread.

I totally agree that a rewrite is risky. It's not something I'd choose to do again, but at the time we didn't really see any way around rewriting the bulk of the database (even if we kept it implemented in Go).

Using Rust and the Arrow ecosystem of projects (Parquet, DataFusion, Flight) meant that there were a ton of things we didn't have to do from scratch. One of our staff engineers, Andrew Lamb, has called it a toolkit for building databases. Thanks in part to his contributions, I think he's right.

For v3 we're supporting InfluxQL natively in addition to SQL. The InfluxQL implementation is actually just a front end on top of DataFusion, the SQL engine we use.

We really wanted to bring Flux along too, but found that it was too difficult in the near term to have it work well with v3. We spent a bunch of time building a gRPC API that Flux uses to talk to v3 (the same thing we have in our Cloud v2 product), but that API was designed with the previous storage engine in mind. It ended up being brittle and performed very poorly.

So at this point the long term supported languages for InfluxQL and SQL, but we're continuing to support Flux for our customers.

We're trying to make the transition from v1 to v3 easier by brining the write and query APIs from that version forward. We wanted to do the same for Flux, but found it was too difficult in the near term. We might be able to do something in the future, but for now we're focused on making core improvements to the v3 engine.

We'll have data migration tools for v1 and v2 into v3 later this year/early next.

So it sounds like you're asking about our use cases. We have customers across almost every vertical. But what's most common are application and server monitoring, sensor data (industrial, rockets, satellites, etc.), and network monitoring.

Metrics is certainly one use case that people pay us for. With v3, we expect that real-time analytics and some more data warehousing types of use cases will become interesting. We always envisioned InfluxDB as a store for observational data of all kinds, not just metrics.

On how we make money, we sell our products. We have at this time:

- InfluxDB v1 Enterprise (a self-managed, clustered implementation of InfluxDB)

- InfluxDB v1 Cloud (Enterprise, but as as single-tenant managed service. We still run this for hundreds of customers)

- InfluxDB v2 Cloud (multi-tenant, usage based, we're running this for thousands of customers)

- InfluxDB v3 Serverless (multi-tenant, usage based v3)

- InfluxDB v3 Cloud Dedicated (single-tenant, resource based pricing)

- InfluxDB v3 Clustered (self-managed v3, clustered database)

We'll have single server versions in the future, but we're a bit off from that. Right now our focus is on continuing support for our v1 and v2 customers, and further developing our v3 products for new customers and customers that want to migrate over.

It's definitely not something I'd do again. I'd warn off anyone from taking this approach. But the early results are looking good and I'm very excited about what this enables for the next few years ahead of us.

To be fair, we didn't drop everything and do a rewrite. Over the last 3.5 years (the length of time for this project), our total engineering team has ranged from 50-90 people. For the first year it was me and two other people. Then for the 2 years following that it was 9 people total.

It wasn't until late last year that we made the decision to go all in on the rewrite and made that the focus of everyone in engineering. And we did that because we had 4 years of experience trying to get v2 and Flux to be successful, with modest results.

Most of the time we were developing this version, we were spending massively more engineering effort on developing v2 or maintaining v1 for our customers.

I still love Go, but I think Rust is a better fit for very performance sensitive systems software like a database.

We'll have to test Chronograf against 3.0, but I think it should just work. Unfortunately, we don't have the resources to continue developing it, but it's all available under a permissive MIT license here: https://github.com/influxdata/chronograf

All versions will support infinite cardinality, including Edge and Community. That's a result of the database architecture.

I expect that one of the use cases for the embedded VM will be to downsample data so I think it's something you could do inside Edge.

I think Edge would be suitable for this use case. It's not that longer time period data won't be stored and queryable, it's just that Edge isn't optimized to make that longer period data queryable very fast (because it's not rewriting the files to reorganize them for that). Ultimately, we'll have to test once it's ready.

Telegraf is still very much alive and well. With v3, we realized the scope of what we were trying to do in v2 was simply too large. We're putting our focus back on the core of the database. Having an embedded VM (it'll be either Python or Javascript) will eliminate the need for Kapacitor in the stack, which had its own programming language and execution engine.

Ultimately, we're trying to narrow the scope of what we're developing and letting other tools that people already know and use have first class placement in the InfluxDB stack.

V3 will support the V1 API and there will be data migration tools for both v1 and v2 into v3. For our customers, we continue to support v1, v2 and v3 and will for quite some time.

With v3 we're bringing the v1 API into it so that people will be able to interact with it as if it were a v1 database. There will be data migration tools from both v1 and v2 into v3. We would have loved to bring the v2 API along, but ultimately the API and the Flux language surface area were far too large.

We tried to do too much all at once with v2. Our goal with v3 is to bring the focus back onto the core of the database.

I said a bit of this in our community Slack channel earlier today, but I think it's worth repeating here:

I really would have loved for us to have more time developing Flux as a language, but we spent all of our time trying to optimize its performance. As a result, we weren't able to make changes to it that I think would have helped people understand and use it.

We pushed on it for 4 years and were told again and again by new users and people that chose other tools that they didn't want to adopt Flux and found it difficult to use. I realize that this wasn't universal and we had some real fans. But we still weren't able to bring the performance that most of our users required. At the end of the day it was just too much to try to do in addition to creating the core database and all the managed, hosted service around it.

I was told once quite some time ago that while Flux made the impossible things possible (i.e. you could do things you actually couldn't do in other query languages), it made the easy things hard. I think this was a correct assessment and that it's what caused the getting started experience to be so poor.

We knew this and wanted to improve it. I brainstormed with Nathaniel (primary Flux author & creator) around 3.5 years ago to make changes to the language. We did a lot of work and documented the whole thing: https://github.com/influxdata/flux/blob/master/docs/new_flux...

But we couldn't ever get to that work. We shelved it because we were constantly fighting performance and reliability fires.

With version 3 we built around an existing query engine (Apache DataFusion) and it's getting developed not only by us, but developers around the world. We think this will have huge benefits over time that lead to a faster query engine that has more features and is more reliable.

The git history was preserved. There aren't really any external contributions to speak of. We've been the sole developers of IOx/InfluxDB 3.0 over the last 3.5 years.

Hi, post author and founder of InfluxDB. Happy to answer questions here.

I went here in November 2001 right after it opened. It was my first time in NYC. I loved it, thought it was so cool. After that trip I decided I’d have to live in this city at some point. Moved here in 2005 and been in NYC ever since. Remote Lounge always reminds me of how magical and full of possibility the city felt (indeed still is!)

It's a terrible situation and we failed on many levels on this one. We will improve our process from here and conduct a more full post mortem.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influxdata.com.

It's built around the arrow-rs library, which we've contributed to significantly: https://github.com/apache/arrow-rs

The open source build will be using the libraries of what we have in our production system, but it'll be a single server release, which isn't something we're running in production. So while many components are things we've been running production for a while, the server process as a whole will be new.

For new users we are suggesting they use either InfluxQL or SQL. Both are supported natively in 3.0 and we'll continue to support them. We were able to bring InfluxQL support because of its similarity to SQL. We were able to build an InfluxQL parser in Rust and have that converted into DataFusion query plans (the SQL engine we use).

Flux is an entire language and runtime so we weren't able to implement it natively in Rust. We'll continue to support it for our customers, but the path forward long term is InfluxQL and SQL.

We also submitted a FlightSQL plugin to Grafana that works with InfluxDB 3.0. So either the InfluxDB 1.x plugin using InfluxQL or the FlightSQL plugin work.

InfluxDB 3.0 is built around a columnar query engine (Apache DataFusion) with data stored in Parquet files in object storage. Eliminating cardinality concerns was one of the top drivers for creating 3.0. I mention some of the other big things we wanted to achieve in some other comments in this HN thread.

InfluxDB 3.0 is optimized for ingestion performance and data compression, paired with a fast columnar query engine. So we can ingest with fewer CPUs, less RAM and reduce storage cost because it's all compressed and put into object store. And we support SQL now (in addition to InfluxQL) with fast analytic queries.

We don't have open source releases yet (that's for later this year), but we have it available in the cloud as a multi-tenant product or dedicated clusters.

We'll be releasing an alpha of InfluxDB 3.0 open source later this year. I'm actually personally on the hook for this one while our engineering team is focused on our cloud and on-premise commercial offerings.

So it's planned, but it's going to take a little time.

This is a difficult one to answer succinctly, but I'll leave some quick thoughts.

One of the things that made this tricky is that we weren't just replacing some small system with a single API. We fundamentally changed the underlying architecture of the database and built it around an entirely different paradigm for querying. This is the result of building it around a columnar query engine with a database architecture designed for the cloud and object storage.

So we made a bunch of changes all at once. We didn't start out this way. We wanted to enable some things in the DB like infinite cardinality, tiered data storage, SQL capabilities and a bunch more. When we saw all that, I knew we'd be rewriting the database one way or another.

This was in early 2020. And I figured if we were going to look at some significant rewrite, I'd probably want to do it in Rust. But rewriting your core in a new language is a highly risky endeavor. Honestly, if you can figure out a way to do it iteratively, that's what I'd recommend. A big bang rewrite is the worst possible thing you can do. And it's super stressful.

But... I didn't see a way around that. So we started small with me and one other person working on it starting around March of 2020. Then we added another team member in May (hey Andrew). The three of us spend the next 6 months treating it as a kind of research project. We evaluated building it around existing database engines (like DuckDB and Clickhouse) and looked at what tools we'd want to use.

By August of 2020 we'd settled on building it in Rust with Apache Arrow, Apache DataFusion, and Parquet as the persistence format. I announced this crazy plan in November of 2020 at our online conference and said we were hiring.

Over the first 3 months of 2021 we formed a team around it of 9 people. Everyone else in the company was still focused on everything else we were doing. So the majority of our engineering efforts were focused elsewhere. I think this was critical. Actually, it was quite difficult to have 9 people this early in the project. We hadn't originally planned to scale up that quickly, but we had a flood of great people interested in joining the project (new hires and internal transfers) that we decided to go for it.

Over the next few years we kept this small group working on the new DB while everyone else was working on previous versions of the product. In mid-2022 we were far enough along to bring up the database alongside one of our production environments and start mirroring workloads onto the new DB. This was critical over the following 6 months or so.

We started getting more people from the engineering team looped into the effort in the 4 months leading up to the first launch.

Starting with a small team and scaling up as you get farther along is critical, I think.

There's so much more I could probably write about this, but I'll leave it at this for now :)

The differences in 2.x and 3.x are quite significant. The 3.0 database was a ground up rewrite in a new language (v1 and v2 were in Go, v3 is in Rust).

InfluxDB v2 was all about the Flux language and a much broader set of API capabilities along with an integrated UI.

For 3.0 we focused on the core database technology. We were able to bring the 1.x APIs forward, which means 3.0 supports both InfluxQL and SQL natively. We were only able to add Flux support through a separate process and a lower level gRPC API that the two use to communicate.

The underlying database architecture is also completely different. v1 and v2 are essentially an inverted index paired with a time series store. v3 organizes data into larger Parquet files and pairs that with a columnar query engine (Apache DataFusion) to execute fast queries against it.

Me or someone on our team should probably write a detailed post about the underlying database architecture to highlight the differences between the versions.

We built 3.0 mainly to accomplish some things that we were unable to deliver in v1 or v2: * Unlimited cardinality * Tiered data storage * Fast analytic queries * SQL compatability * Bulk data import and export (coming soon to v3)

Then there are the systems architecture changes we made highlighted in this blog post. v1 InfluxDB was a monolithic database that had all these components in one. The v3 design allows us to scale ingest, query, and compaction separately, which is something that kept coming up in larger scale use cases.

Hello, one of the authors here, happy to answer any questions!

Likely the 3 founders have 12% each (after a 4 year vest) and the remainder (21%) goes to the option pool.

We're still figuring out what will be in InfluxDB 3.0 open source. For now we're focused on our cloud offering where we can iterate and debug more quickly. We expect that the open source will fill a different need than the cloud product, but we're still not sure. The Clustered and Edge products mentioned will be commercial offerings.