There is also Apache Arrow that uses a similar use case. Maybe this is worth considering: https://arrow.apache.org/docs/python/memory.html#on-disk-and...