Hacker News Reader

Comment by nicobrevin | Hacker News Reader

It would be nice if the hdf5 people could add better support for concurrency, a la https://www.hdfgroup.org/2020/11/webinar-enabling-multithrea...

I can't imagine there'll be a queue of people wanting to implement that for fun on their weekend

View on HN

Mostly sounds like a bad idea.

HDF5 is on the way out anyway.

What’s the replacement?

HDF5 does so much there's really not a single replacement. The most prevalent is probably JSON from what I've seen, but SQLite, (C/T)SVs are also fairly popular. There's a ton of application-specific serialization formats too.

None of those (JSON, SQLite, CSV) are remotely appropriate for addressing HDF5's niche of storing large blocks of numbers in a manner that can approach their native space / perf characteristics.

Yea, HDF5 is great at what it is great at. The problems however come when people try to use it as a general data storage format and pseudo database.

Right, it does get abused, but I don't see anyone coming for its niche and I don't see its niche evaporating.

Lol yeah, I always reach for paperback books now since they're a great modern replacement for my Kindle.

On a more serious note, DuckDB is actually a pretty fantastic replacement for SQLite for some data analytics tasks. Hdf5 is still great for numerical data in this space though. I would welcome other options like the SQLite/DuckDB scenario for matrix data.

If there is any replacement in the HPC world (and I don't really see indications for that), it's Adios2.

H5AD

Arrow.

It's absolutely not "on the way out". Lots of very slow moving, very deep pocketed organizations working on very long time horizons are heavily invested in HDF5. The same can't be said for any of the flavor-of-the-week ML "cloud-native" file formats.

There is an MPI version of HDF5 no?

Yeah, but no multi threading concurrency, only multi process. Not even parallel reads. There's a giant mutex around the call to read data from a dataset and my colleague's take on it was that it would be months of work to unpick all the state and make even reads thread safe.

Reply on news.ycombinator.com