Hacker News Reader

What are the new file format initiatives you're referencing here?

This solution seems clever overall, and finding a way to bolt on features of the latest-and-greatest new hotness without breaking backwards compatibility is a testament to the DataFusion team. Supporting legacy systems is crucial work, even if things need a ground-up rewrite periodically.

View on HN

Off the top of my head:

- Vortex https://github.com/vortex-data/vortex

- Lance https://github.com/lancedb/lance

- Nimble https://github.com/facebookincubator/nimble

There are also a bunch of ideas coming out of academia, but I don't know how many of them have a sustained effort behind them and not just a couple of papers

Lance (from LanceDB folks), Nimble (from Meta folks, formerly known as Alpha); I think there are a few others

https://github.com/lancedb/lance

https://github.com/facebookincubator/nimble

I’ve been excited about lancedb and its ability to support vector indexes and efficient row level lookups. I wonder if this approach would work for their design goals and still allow broader backwards compatibility with the parquet ecosystem. Have been intrigued by Ducklake, and they’ve leaned into parquet. Perhaps this approach will allow more flexible indexing approaches with support for the broader parquet ecosystem which is significant.

Reply on news.ycombinator.com