If anyone is interested in similar challenges, but in cryo electron microscopy, where the datasets are dozens of TB, this is a good paper discussing the file formats, header organization, and bit-depth required for proper resolution:

https://pubmed.ncbi.nlm.nih.gov/35724904/

What I can say is that TEM data management is unmitigated hell, and thank God for that author (Ludtke) and his software (EMAN2).

Biologists are unsurprisingly awful at data management.