Pulsar: Real-Time Analytics at Scale | Hacker News Reader

Pulsar: Real-Time Analytics at Scale

ebaytechblog.com

54 pointsmreddy14 comments

Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second response time?

From a glance, you have to define the queries before. Are you using anything currently?

What is your insert/update/delete rate? If low, maybe look at postgresql in-memory column-store https://github.com/knizhnik/imcs/

Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app.

Also curious.

I specialize in an in-memory analytics (business intelligence) platform called "QlikView" or "Qlik". It operates on top of RDBMSs/operational databases, so it batch processes data on a daily or hourly basis usually.

If you are already familiar with QlikView, then check out Google BigQuery, you already have a connector for it.

Could just use a search db (e.g., elasticsearch) depending on the structure of your data and the aggregations you need.

For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.

I would go to with elasticsearch, or if you reach terabytes/petabytes of data then look into Google BigQuery there you would get a constant 5-10 seconds response time for those high volume data. BigQuery is a WORM technology, so an append-only system might help you, especially in event-driven systems.

BigQuery is interesting. Thank you for the link. I am trying to determine if large quantities of data like this can be analyzed with sub-second aggregation time, targeting a 2-3 second total response time to the user, including network latency. The hypothetical data set would not necessarily be event driven, but of course it is useful to have lower data latency. The aggregation speed is the biggest performance concern for me.

The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).

Looks interesting, a bit similar to Apache Storm https://storm.apache.org/

Any insight on how this one compares to Netflix's Druid? Really curious.

Anyway Druid is by MetaMarkets, Netflix is one of the users

Thanks, I stand corrected.

They complement each other. Druid is "only" a real-time data storage for events and fast aggregation. This is a complex event processing engine, which would store its stuff in druid.

Reply on news.ycombinator.com