I've been looking for a solution to the same problem.

So far we've been using a single server for storage, and developers rsync whatever they need locally. A few million images. Training is usually done locally, but when we do use cloud training, we upload just the dataset we need to S3 and use EC2.

We're a small team, and currently considering moving to a cloud-first infrastructure. The idea is to store each image in S3, and all metadata (annotations etc) in Postgres or something like that, maybe using Postgres's JSON/JSONB feature.

I'd appreciate any thoughts and pointers on handling datasets with a few million images.