Ask HN: Any good reads, articles or books on file migration?

4 pointsbosch_mind4 comments

I’m looking to build a tool to migrate files between cloud providers (ie blob storage, s3, etc).

I’m looking to learn best practices for doing so. The volume is a few petabytes but can be done incrementally.

I was thinking some queue that handles N transfers concurrently running in a k8s cluster. Queue I was thinking Redis or Kafka. However, I’d love to build something nice and resilient

View on HN

Don't have any articles handy (beyond the various cloud providers' offerings which you can find with a quick "s3 to s3 migration" search)

But the egress and ingress fees, added-onto the middleman position your tool would be occupying sound heavy :)

Big company wallet :)

why not aim for something that doesn't hurt the "Big company wallet" :)

This tool already exists and is fairly standard. No extra infra required.

https://rclone.org/

Reply on news.ycombinator.com