InfluxDB 2.0 Alpha and the Road Ahead
influxdata.com
influxdata.com
That said while TickScripts are not exactly a pleasure write, I wouldn't say a new language is pretty high on my wantlist. For example, as the blog post states:
> We also need to add features for backup & restore, bulk data import and export, and data deletes.
I'd take any of those over a new query language any day now. Especially the restore part is at the moment nearly completely non-existing after backup format change in InfluxDB, so good luck if you happen to run into data loss or just want to move around data in general. Same goes for data deletion outside of whole measurements.
I am hopeful that those features arrive before a 2.0 launch and hopefully in some sort of backwards compatible way (or at least ported to Influx 1.x).
Interesting to see that we at VictoriaMetrics also pay attention to these features. Backup and restore already works - see below. Bulk data import and export plus data deletes for arbitrary metrics selectors will be available soon.
It would be great if InfluxDB 2.0 would have instant snapshots as VictoriaMetrics already has - https://medium.com/@valyala/how-victoriametrics-makes-instan... .
Also glad to see that the future InfluxDB versions will support Prometheus exposition format and PromQL :)
What does the data model for time series look like in 2.0? Mostly the same as 1.x, or has that gotten more flexible as well?
Once the planner gets the data to the Flux processing engine it views everything thing as a table of data with columns and records. So it’s much more flexible in how we can represent data.
> With InfluxDB 2.0 we support both push and pull models out of the box. Eventually we’ll also support querying via PromQL out of the box
Any HA features? Sharding to look out in 2.0? Or is the general idea to set streaming relays of influxdb tsm's and treat HA as an L7 proxy routing problem (shadow metric traffic using envoy for example). How do people handle this in their production setups. Curious to know.
It would be cool if the query engine could talk to multiple shards spanning multiple machines for dealing with high cardinality series.
So you could certainly layer in your own HA solution. We’re still working out what if any clustered for federated features will exist in open source.
(I'm not affiliated with Influx, but I strongly believe developers need to get paid with money.)
https://github.com/gravitational/monitoring-app/blob/master/...
I looked at flux, and it seems pretty compatible with TICK script, although I don't have a clear understanding on how to make it easy to write alerts/queries and debug them yet.
I would be curious to try out REPL:
> We plan to provide a flux command line program that exposes a REPL and talks to various data sources.
Especially interested to see how easy it would be to edit and troubleshoot a multi-line transformation query like this one in it:
cpu = data
// only get the last 5m of data
|> range(start: -5m)
// only get the "usage_user" data from the _measurement "cpu"
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user")
e.g. your flux example above could have been just:
let cpu = data.range({start: "-5m"}).filter(func(r){return r.measurement == "cpu" && r.field == "usage_user"})
https://www.influxdata.com/blog/why-were-building-flux-a-new...
and it brings some valid points on Flux vs SQL, however I would be interested how would it compare as Flux vs Javascript or Flux vs Lua :)
Either way, I'm hoping this is "The Release" where things go consistent. They've pivoted several times, and it's all been good, but we do need a couple of years of consistency before we can make long term investments into Flux.
One of the reasons we didn't go with that is because we didn't want a separate query vs. scripting experience like you have with SQL engines that embed programming languages. We wanted something that felt and looked seamless.
We chose not to use Lua because it's not a widely known language. Instead, we chose to create a language that looks and feels much like Javascript, which I think is the most widely used language today. However, we wanted to limit the scope of it and to add new syntax over time to make frequent tasks easier to express.
Any time you create a new API or library, you create new surface area for a developer to know. A language is no different than this. Some languages are also much easier to learn than others. For example, I found Go a language that I could pick up the basics in very quickly, while Rust is something that I'm still learning over months of effort. I like both languages, but their learning curves are very different.
I think Flux is quite easy to pick up for many programmers, although I'll have to see questions and have countless interactions with people trying to learn it to prove that out and make adaptations over time.
Finally, one goal with including the UI in InfluxDB 2.0 is that as we improve it, most users won't have to learn Flux at all. They'll be able to accomplish what hey want by just clicking around the interface. We're not there yet, but it's what we aspire to. And we want to have control over the language to build it in conjunction with UI tooling to automatically manipulate it.
This makes sense, in this case why not take strict subset of Javascript?
> Finally, one goal with including the UI in InfluxDB 2.0 is that as we improve it, most users won't have to learn Flux at all. They'll be able to accomplish what hey want by just clicking around the interface.
This will definitely will be helpful, and auto generation makes sense, however in our use-cases we treat Kapacitor alerts as a code that is reviewed.
However, there are other things we'll probably be adding to the language over time. Also, if you're only allowing a subset of JS, is it actually JS anymore? I'd imagine that some JS programmers could get frustrated when things they think should work (because they're valid JS) don't because they're not in the subset.
But say we went with that. Then we'd need to adopt an existing JS engine (most likely written in C++) and then modify it based on our needs. And then integrate that into our broader Go codebase. All of that seems a bit less than ideal.
Ultimately, I really do feel that if you're someone that knows Javascript, you can learn the elements of the Flux language in less than an hour. The bigger learning curve is the library of functions and the API, which would exist regardless of what language we chose.
I think the proof will come over time based on what kinds of things we enable in the language and the platform. In the near term I expect a large number of totally reasonable people to question the choice of a new language. After all, that's probably the rational response. But as we improve it, add to it, refine it, and improve the developer and user experience, I expect to win more converts. Developers pick up new tools because they enable them to get their jobs done faster. Ease of use, speed of development, and productivity are our guiding lights.
This is definitely not an easy choice to make and writing a new language interpreter in Go seems a way easier approach initially.
One thing that Go helped me to understand though, is that the language does not matter so much as the tooling around it - debugging, compiling and support around the language are way harder to achieve than writing a language parser.
Thanks for sharing your thoughts on this design choice, I think this discussion is very relevant for a couple of reasons:
There are other Go projects that are taking the same approach to achieve simplicity and not picking the existing language, like OPA [1]. Others, like Helm are picking Lua [2], so there is clearly a problem the Go and infrastructure community is facing and split in the way people are approaching it.
We've faced the similar dilemma with Teleport, as we are designing our extensions system. My original plan was to use Lua, however after discussions with the team we settled on GRPC with Go [3], trading expressiveness/simplicity and freedom of lua/js languages in favor of industrial features Go runtime and GRPC provides out of the box.
For smaller extension plugins we decided not to create a new language, and ended up with interpreted subset of Go [4].
I wonder if there is a place for some subset of javascript or typescript that is fully interpreted by Go, with native extensions for debugging to be used by the community.
[1] OPA policy agent https://www.openpolicyagent.org/ [2] Helm 3.0 Lua plugins https://github.com/helm/community/blob/master/helm-v3/005-pl... [3] Teleport Plugin Design Document https://docs.google.com/document/d/1sPXXxx03P8VXWy-YD5w190g7... [4] Subset of Golang https://github.com/vulcand/predicate
And can data points be incremented instead of the current field-replacement crap when you get new points with the same tag set?
- Significant TSM encoding and decoding performance improvements.
- The TSI index will be on by default.
- Queries that use the same tag key/value filters will be answered from the index more quickly using an LRU cache.
- Field keys will now be indexed in 2.0, making filtering/grouping on field keys more efficient.
- Improvements to how series are extracted from the index, and points data from the TSM engine, which helps with memory performance for queries.
- Significant performance improvements to measurement deletion.
Can you elaborate on that?
I want to use influx to store _statistics_ not _events_. Basically, my data points are tag-sets and counts.
There are several ways to achieve this; for example, you can send the events to influx and have continuous queries to gather the statistics. That doesn't work well when you have a lot of events, and where they arrive out of order and at high latency, etc.
So what you typically end up having to build is a stats thing that sits in front of influx and tracks the counts of events with particular tag sets in particular time buckets, and then keep uploading these to influx.
And there are two ways to do that:
1) you are not stateful and you keep uploading deltas and incrementing the nanoseconds to avoid data-point collision; you can then get the data out of influx with sum() on the fields and grouping by whatever the time bucket is. I tried this and influx grinds to a halt eventually.
2) you are stateful and track the totals outside influx, and keep uploading a newly-written data-point to overwrite the fields for that bucket in influx. This is much less data in influx and much easier to query, avoiding sum() etc. Its like I end up with something in front of influx doing what I want influx to do.
What would greatly simplify life is if the line format which looks like this:
measurement,tag1=x,tag2=y,tag3=z f1=total,f2=total timestamp
could look like this:
+measurement,tag1=x,tag2=y,tag3=z f1=delta,f2=delta timestamp
and in the second case, where the line is prefixed with a + sign, influx knows to add the fields if the data-point collides with another rather than overwrite them.
This would mean that people trying to store statistics in influx could add to those statistics statelessly. A massive simplification.
I've had other problems, like I have way more than 1M series. Its painful. My influx boxes hit iowait far too often, which is weird because the boxes have more RAM than the total dataset.
After InfluxDB 2.0, what applications would I have to run to be completely tick-compatible?
I ask this because I would rather not have to run a DB on every server because Telegraf was fully merged into InfluxDB.
Are there any plans for a more unified method of performing continuous queries - so that we can query high granularity and older, downsampled data at the same time?
That would be a killer feature for me.
I agree with the folks below suggesting you use javascript as the language instead of reinventing your own.
We used to use InfluxDB to store our perf data where every data point contained a timestamp (thus we thought influxdb was ideal for our use case). But, soon we wanted to group, filter, sort by various dimensions and it lead to performance issues.
Bringing it all in-mem and using pandas to do that was very slow for us. Also, creating indexes for so many columns didn't seem like a good idea.
We switched to postgres and the decision has served us well so far. I just want to understand if influx isn't suitable to our kind of use-case or we used it incorrectly.
Um. Yes. Wait, there are people that use influxdb _without_ kapacitor alerting??? Someone name any commercial or open source equivalent, b/c I'd like to know in case they kill it.
AM looks ugly as hell („designed by engineers“), but it‘s damn rock solid and versatile.
Tried Influx in the first days, but it was slow and buggy and they changed their paradigms every other version. Well, looks like they‘re doing it again...
People who use InfluxDB for reporting, not for alarms.
Our reporting team uses InfluxDB-sourced data to send customers reports on things such as bandwidth utilization and performance statistics, and we also have some reports and dashboards for internal purposes. And honestly, we use Zabbix for a lot of internal metrics too. For simple things, it's much easier for me to just set up an item and a graph in Zabbix than it is to stand up a "cron" job to farm data into InfluxDB ("cron" in quotes because it's not literal cron but a Jenkins-based system).
For actually monitoring devices for problems and generating alarms, we use Zabbix for internal systems and Netcool for customer-facing systems.
(for the record, I'm an NMS engineer at a mid-sized telecom)
Good content, it's a shame it's not more popular.
If you really need the HA, you can start spending on the enterprise edition. Or move to another storage.
The other thing I never figured out in the current version is how to write the following query. I store samples like (device, network, direction) -> packet count. I then want to know how many packets were sent across the network in total.
With monitoring systems I've used in the past (internal to my former employer), this was easy. You would do the delta calculation at the lowest level to convert device packet counters to number of packets sent in the last time interval, which varies randomly because samples are not necessarily arriving at discrete intervals. Then you would do an align, to bring the randomly-added sampling times into alignment across all the "streams" (which is a unique (device, network, direction)). Once the data is aligned, you can then do a group by to get rid of a certain tag, like device (and just have (network, direction) -> total packets sent in the time interval).
With InfluxDB, I can't figure out how to do this. It has the group by time concept, but not alignment, so I can't write a query that will work. I ended up just computing the deltas before inserting the item, at which point everything works fine.
I have not tried the same query with Prometheus, but I suspect it would work like I expect, as it seems very heavily inspired by a certain internal monitoring system I am most familiar with.
For the query you mention, you can do that in 1.x using a combination of group by time, fill and subqueries. In 2.0 you can do this in Flux using similar operators. More complex possibilities for interpolating missing data is also in the works.
Solving more complex query processing like what you mention is a specific reason why we started developing Flux. We couldn't figure how to address some of the more advanced query feature requests in the old language so we started with something new.
In addition to giving Flux more functions, flow control and more power, we also will be adding more out of the box functions and syntactic sugar to make some of these more advanced queries possible without being overly verbose and complex. Our order of operations for developing the language:
1. Make it powerful 2. Make it easy 3. Make it fast
So it'll take us time to get to all of our goals, but I think we've laid down a pretty good foundation.