Hacker News Reader

Comment by riboflavin | Hacker News Reader

Dremio helps with a lot of this, particularly the speed aspect – uses Parquet as well as Apache Arrow. (I work at Dremio.) Speeding things up: https://docs.dremio.com/acceleration/reflections.html

View on HN

Dremio quickly becomes useless with MongoDB given that for a while it's not been possible to join data from two MongoDB collections by their object IDs. Last time I checked, Dremio mangled the id into some string that can't even be matched to the same id on a separate collection.

I had data in PG and Mongo, but couldn't join it together. I asked about this on the forum, was told it's a known issue; and it seemed to end there.

I resorted to doing my analytics by hand in the end, MongoDB's aggregation framework is good enough. Create views from aggregation queries, and it becomes easier

The downside is that one needs a business license to use the BI connector.

> The downside is that one needs a business license to use the BI connector.

Have you looked the postgres mongo fdw[0] before?

[0]: https://github.com/EnterpriseDB/mongo_fdw

I’m only familiar with speeding up Parquet - it looks like you’re mainly sorting or partitioning the data into different views so that you can choose the best view format at runtime based on your desired query or aggregation? We’ve seen this increase speeds by many orders of magnitude, so I wouldn’t be surprised that this creates meaningful speed-ups when done automatically (Which is cool!).

Random aside, how do you handle the consistency problems that can occur when you have multiple views when doing deletes?

Reply on news.ycombinator.com