Hacker News Reader

Comment by mgaunard | Hacker News Reader

Mostly sounds like a bad idea.

HDF5 is on the way out anyway.

It's absolutely not "on the way out". Lots of very slow moving, very deep pocketed organizations working on very long time horizons are heavily invested in HDF5. The same can't be said for any of the flavor-of-the-week ML "cloud-native" file formats.

What’s the replacement?

HDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.

None of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.

Yea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.

Right, it does get abused, but I don't see anyone coming for its niche and I don't see its niche evaporating.

Lol yeah, I always reach for paperback books now since they're a great modern replacement for my Kindle.

On a more serious note, DuckDB is actually a pretty fantastic replacement for SQLite for some data analytics tasks. Hdf5 is still great for numerical data in this space though. I would welcome other options like the SQLite/DuckDB scenario for matrix data.

If there is any replacement in the HPC world (and I don't really see indications for that), it's Adios2.

H5AD

Arrow.

Reply on news.ycombinator.com