Hacker News Reader

I was looking for something local, a library or sth like that. I don't want to run other processes just to query that memory. Spark is enormous and slow in comparison with local in memory solutions.

View on HN

Gotcha. Just some confusion around what you meant by in-memory OLAP engine. It seems that many storage format libraries get integrated into other platforms that bring the other supporting features (and the other undesirable heavyweight things you didn't want) with them.

With a little bit more poking: Apache Calcite?

Calcite is more a query optimization framework. Actually, I think parent has a good question, which is, how to use these tools, formats, etc. from something simpler than an entire full blown distributed technology - and I think the answer to that is, there aren't too many technologies out of the box that do that right now.

I would think maybe extending SQLite to use columnar storage would be one possible path that might make sense.

There are columnar DBMS engines you can embed like MonetDB, but I personally haven't seen a lightweight query engine sitting right on top of Parquet/ORC, or implementing Arrow for in-memory columnar. You can get Apache Spark up and running pretty quickly on a single node, but I haven't seen anything even easier than that.

Reply on news.ycombinator.com