marsokod

One of the problem with NASA (and other companies/agencies of the sort) is that projects take a very long time, at least 10 years and usually 20 years. You cannot fire the guy who officially did the estimate at the beginning: he is not on the project anymore, most probably not in the same company, maybe retired, and usually has grown to a position much more powerful.

Underestimating on the contrary leads you to not getting funding for the projects you are responsible for, and that is a nice way to not get a promotion.

As in Python, it is easier to ask for forgiveness than permission.

I developed an archive system for our spacecraft TM/TC using HDF5 with blosc-lz compression for speed reasons (and I plan to move to Zstd in the future). One of the main issue I have seen in the industry is the difficulty to upgrade the hardware so we have to design something working great on regular hard drives and with minimal RAM requirements.

While HDF5 is good when you manage the whole system, it is always tricky for sharing data with other people so we are also using SQLite. We lose the compression but it is very easy to share and people are more familiar with it.

You can also check out jirafeau, quite light and easy to install: https://gitlab.com/mojo42/Jirafeau and demo site at https://jirafeau.net