I think this is where Array Databases shine, like https://github.com/TileDB-Inc/TileDB