Open-Sourcing Pinot: Scaling the Wall of Real-Time Analytics

I came here to ask for a comparison between the two. I'll add the request here instead instead of starting a new top level thread.

Both look really interesting!

The architecture is extremely similar.

Has anyone tested query volumes on Druid.io? We're thinking about using it in production (may now also consider Pinot), but we're concerned about the lack documentation around production load query volumes. Really like that LinkedIn uses Pinot at runtime.

I'm biased in that I had the honor of writing the first line of code of Druid, but from looking at what I could find on Pinot it looks like architecture-wise, we share a lot of similarities. At a minimum, the thought behind how to scale and how to ingest data appears to be the exact same. Druid clusters are presently ingesting tens of billions of events per day (with a 50b number from Metamarkets in the venturebeat article) and housing trillions of rows. From a high level architecture, I don't see a reason Pinot couldn't as well (when they are the same, it's kinda hard to say that one can't handle what the other can ;)).

On the query API-side, it appears like Pinot exposes "PQL" which is a SQL-like query language. In Druid, for better or worse, we use JSON-based HTTP calls. Once you are using a client for querying in a programmatic manner, the difference doesn't really matter, but it is true that something like PQL makes it simpler for humans to write their own queries.

For management of distributed state, it looks like Pinot depends on Apache Helix, which is an interesting framework. This was the first time I had heard about it and, I really would've liked to have something like that to build on when we started Druid. We instead had to implement it all ourselves. Helix sounds like it provides all the hooks to enable the various levels of integration that we have needs for in Druid, so I can only imagine that there is little difference in capability there. My biggest fear is that the majority of operational issues that we run into with Druid is around zookeeper integration, losing an entire ZK cluster is painful (and does happen) and from my read of the Helix docs, that would likely cause a loss of all shards of data as well (it looks like they are persistent znodes?).

In terms of storage format, Druid enforces bitmap indexing and the Pinot docs appear to indicate that they have multiple types of indexes available, which is pretty cool. It looks like Pinot doesn't have a way to plug in user-defined column formats, though, so it would likely be require a bit of extra work to extend the system to be able to handle more complex types of aggregators like sketches. And, really, enabling sketching extensions is one of the most important features of any "scalable realtime processing" tool, in my opinion.

On the ingestion side, it appears that Pinot reads data from kafka in real-time and then restating from batch, which is the model we've employed in Druid for the last 4 years, so we know it works. I'd love to know how Pinot guarantees high availability and replication of the real-time stream, there are many ways of doing that (over the years, we've implemented 3 different ways in Druid and are still not particularly happy with any of them).

Anyway, I still haven't had a chance to play with Pinot, so this could be completely off-base, but those are my initial reactions.