Yes, the amount of storage can be an issue, but I want to stay below 500GB of hot-data. One bottleneck is network traffic to copy updated files after each weather model update.
My binary files are just plain Float16 files without an meta data. Logically they similar to 3D Zarr files. I know exactly which bytes I have to read and the kernel cache helps a look to keep it fast.
In theory this data does not have to be a file on SSD. I could also use a block storage directly or request data from S3 via range requests.
One hopelessly over-engineered approach would be to use an `AWS EBS io2 Block Express Volume` use `Multi-Attach` and spawn up to 16 API instances to serve data from it.