Ask HN: Nan for HDF-5 Data Format

7 points·Eddy_Viscosity2··6 comments

My question is for those familiar with the HDF-5 data format. From what I can tell, it does not natively support the Not-A-Number (NaN) data type. For example if an array of data values was stored and some of them were NaN then these would be converted to nulls which can then be read in (i.e. misinterpreted) as zeros.

Does anyone know of a standardized or widely accepted way of storing NaN in HDF-5 files?

View on HN

NaN is not a data-type, it is a floating point number -- I've seen NaNs (and Infs) in HDF5 files (as floating point numbers). Some of the accompanying tools do not "do the right thing" with them though (h5diff sees NaNs as equal, for example).

Thanks, its actually good to know that I may be wrong in my premise and that I may in fact be able to properly store the NaNs. For context, I'm trying to convert Matlab data to H5 files.

Doesn’t MATLAB have a built in h5write function to write out HDF5? There shouldn’t be much work at all to convert MATLAB data to H5. Last time I used it I had no issues with the resulting H5 files.

There is a fair bit of built-in H5 functionality in matlab, but the devil is in the details and I want to be sure I am storing the data without any loss of representation including NaN, inf, metadata and so on.

Sure. For what it’s worth, MATLAB has used HDF for its own MAT file format since R2006b. See: https://www.mathworks.com/help/matlab/import_export/mat-file...

I’m assuming if the MAT file contains what you expect and is a v7.3 file, it’s all accessible in HDF5 since it’s already an HDF5 file.

Sounds like an issue with the Matlab interface to HDF5 then, good luck!

Reply on news.ycombinator.com