TrailDB – An Efficient Library for Storing and Processing Event Data

Thanks! I hope we can open-source the rest of the stack now that the core libraries are out.

Btw, I am happy to answer any questions about the project here.

And in Java 9, this embeddable C lib will be usable from within Java given http://openjdk.java.net/projects/panama/ :-D Not that JNR can't do this now, but Panama will be awesome. /end hijack

Could not agree more, biokoda. A C lib with all the bindings offered with TrailDB is absolutely a winner. C, Python, and Go. A happy day for sure.

Thank you so much Ville and AdRoll for sharing this with the world! Your talk on the Trillion Row Data Warehouse in Python was an inspiration for much work I have done since. This library will carry that even further.

The approach here is very different. It's not that window functions are bad, but that the traditional SQL database, or even column oriented database, doesn't have a disk format that is versed to this type of analysis for the high volumes that AdRoll has. TrailDB can filter through 10s of millions of events per second on a Macbook, just storing billions of events in a more traditional DB or warehouse would use far more than the space that TrailDBs use. On top of that it's relatively easy to build visualisations on top of these databases with an interactive UI to do exploration. So the window function being complicated was perhaps just the spark but there are real advantages in speed, efficiency and expressiveness in this type of database.

Its not really about SQL or NoSQL - I believe AdRoll internally has systems that map SQL queries onto TrailDB where its needed.

Its about efficiency. TrailDB compresses the hell out of event streams grouped by some key (typically user). I think it's generally something like 100x - it does so by making certain assumptions about your data. Similarly, when operating on your event sequences, many of those operations can be performed as integer instructions on the compressed data.

TrailDB gives you order(s) of magnitude of leverage. That's the difference between doing analytics on a single machine and a cluster in many cases. There aren't too many other databases that were designed to gracefully handle tens of trillions of events, so its really not surprising that you need a different tool for that job.

Efficient data structures and algos, carefully tuned for one particular scenario. Seems like a good candidate for the one of PostgreSQL pluggable storage engines we will likely see in the future[1]. I hope the data structures are not patented.

[1] http://akorotkov.github.io/blog/2016/04/06/extensible-access...

Pipelinedb.com is a great product and very complementary to TrailDB.

PipelineDB is all about querying streaming, real-time data in SQL whereas TrailDB is great for computationally intensive analysis of historical data using any programming language.

It's not quite the same thing, PipelineDB is a database to execute streaming queries at massive volumes. That is definitely one of the challenges at AdRoll.

TrailDB is more about a different way of grouping and events, and granularly querying and analysing each trail of data. TrailDBs are materialized and stored in S3 typically.

I'd also add that PipelineDB excels at SQL-based workloads where you know the queries you want to run in advance and want to stream large volumes of data into PipelineDB's continuous query engine and store ONLY the results of the continuous queries in PipelineDB's underlying relational database. The main value is in continuous computation and distillation for realtime reporting and realtime monitoring and alerting use cases. PipelineDB isn't designed to do ad hoc, exploratory queries, although it can to the same extent that PostgreSQL 9.5 can. It's designed for scenarios where you know the analytic queries you want to run in advance and where a SQL-based approach to streaming analytics with integrated storage provides value by being simpler than building a custom system in Java / Scala using frameworks like Storm, Spark Streaming, Druid, Cassandra / HBase, etc.

AFAIK Druid is for time series and it's a columnar database. Their format has dimensions and then pre-aggregated fields on those dimensions. Afterwards you can run SQL queries on that data format to get full aggregations in return.

This is a library to read and write a data format that is optimized to give access to granular events and actors within an event stream. For example this could be used to trail all of the events generated by one entity (credit card, cookie, email, account and so on) over a dataset. At that point you can choose what you want to do with it: extract features for ML, train ML directly on raw data, run arbitrary queries for outliers and anomaly detection and what have you.

Interesting, but the open source SkyDB seems to be dead:

https://twitter.com/benbjohnson/status/590626780173721601

Parquet is a columnar format, so it's optimized for aggregations over columns filtered by a set of dimensions. TrailDB data format is optimized for discrete event analysis without aggregation: data points are granularly grouped by their source (for example a user account and actions on a website) and your queries operate on each of these groups independently. No aggregation happens in TrailDB.

TrailDB is optimized for a very specific data model (http://traildb.io/docs/technical_overview/#data-model) which allows it to do compression that you couldn't do with other data layouts.

TrailDB is a C library, Parquet and Avro are not. Depending on your use case this might be a pro or a con.

This is a tool to look at each user individually actually, not aggregated.

TrailDB is designed for discrete events (think JSON objects) rather than continuous metrics (like CPU utilization).

Most time-series databases manage to store a large number of data points by aggregating data over time. Discrete events can't be aggregated. Instead, TrailDB leverages predictability of events to compress data over time.

We were just talking about it earlier today. It should be easy to do with https://github.com/node-ffi/node-ffi

Contributions are welcome :)

Played around with it today. Made some progress here: https://github.com/poynt/traildb-node

Let me know if you have any use cases that are in the API but not covered yet by this library :)

Good question. Judy arrays tend to be really efficient for low-level mappings like the 128-bit -> 64-bit map (https://github.com/traildb/traildb/blob/master/src/judy_128_...) that we use to keep track of UUIDs.

It would be interesting to benchmark Judy against another similar well-optimized data structure.

It is coming soon, sorry for the delay!

Well let's see. Perhaps, processing social media data, analysis of stock market data, looking at weather or other observational data, working with log events, handling IoT data streams, ... and on, and on. If you look at the world, an amazing amount of data use cases reduce to events by time by some other dimension.