Hacker News Reader

Comment by bmh100 | Hacker News Reader

Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second response time?

View on HN

From a glance, you have to define the queries before. Are you using anything currently?

What is your insert/update/delete rate? If low, maybe look at postgresql in-memory column-store https://github.com/knizhnik/imcs/

Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app.

Also curious.

I specialize in an in-memory analytics (business intelligence) platform called "QlikView" or "Qlik". It operates on top of RDBMSs/operational databases, so it batch processes data on a daily or hourly basis usually.

If you are already familiar with QlikView, then check out Google BigQuery, you already have a connector for it.

The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).

Could just use a search db (e.g., elasticsearch) depending on the structure of your data and the aggregations you need.

For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.

I would go to with elasticsearch, or if you reach terabytes/petabytes of data then look into Google BigQuery there you would get a constant 5-10 seconds response time for those high volume data. BigQuery is a WORM technology, so an append-only system might help you, especially in event-driven systems.

BigQuery is interesting. Thank you for the link. I am trying to determine if large quantities of data like this can be analyzed with sub-second aggregation time, targeting a 2-3 second total response time to the user, including network latency. The hypothetical data set would not necessarily be event driven, but of course it is useful to have lower data latency. The aggregation speed is the biggest performance concern for me.

Reply on news.ycombinator.com