I can't figure out if one of those three methods can do what I want. I want to include a multi-valued lookup table so given a string id I can find the 0 or more corresponding records. (I can handle single-valued lookup tables, if multi-valued isn't available.)
I looked at Parquet, but it doesn't seem to handle id->index table lookups. Or 256 byte fields. Do any of Zarr, HDF5 and TileDB offer a better fit?
Background: it's hard to compare molecules directly. Instead, there are hashing methods to turn a molecular structure into a fixed-length bitstring of length nearly always between 166 and 4096 bits in length.
Bitstrings are easy to compare using Jaccard similarity, which gives a decent-enough and very fast way to compare molecular similarity. With help from another HN user (who I paid), we were able to get the performance to within a factor of 2 of the theoretical memory bandwidth.
Going faster requires compressed RAM->uncompressed cache transfer, better indexing, and distributed computing. I don't know how to do it using one of the existing Big Data formats, and developing my own format is hard.
My FPB file format is a FourCC format with several chunks. One chunk stores all N fingerprints, cache size aligned. These are ordered by popcount, so in principle some should be easily compressible - I haven't tried, and can't figure out how BLOSC is supposed to be used.
Another chunk stores the corresponding fingerprint identifiers, UTF-8/non-fixed-length strings, in the same order (fingerprint[i] corresponds to id[i]).
A third chunk stores a multi-valued hash table so I can look up fingerprint(s) by name, instead of by index. The hash table is a variant of the djb's "cdb" hash table. This is a 32-bit hash, which means for large data sets I get many collisions. (I designed it for <100M entries. Some people now want to handle >1B entries.)
I'm hoping to use an existing format rather than roll my own. Again.