[1] - https://docs.aws.amazon.com/snowball/latest/ug/create-export...
[2] - https://help.backblaze.com/hc/en-us/articles/360001918654
If we were to consider another large migration like this, physical media would probably be the way to go.
So, snowball works in a lot of area, but like so many AWS products, it works if you adapt to it.
pigz/scp/zstd works extremely fast in line.
In your case you're pulling from S3 to another object store.
I moved ~1PB from one S3 region to another. "Why not use replication," they asked. That only works if it's turned on when you upload the object - another fine-print 'gotcha' in the easy AWS service. Then you get into rate-limits. In 2010 I asked AWS if I could spin up 1000 servers to test something - nope - elastiticy at that level is for the big boys.
Now I work for a large cloud company and we still run into elasticity.
To move the 1PB from one S3 region to another we spun up hundreds of spot instances (oh, we were compressing and glacierizing it too) and built a perl/mysql batch job "s3 get | zstd | s3 put" process and parallelized it. One thing nice about S3 is it pulls the md5 hash - unless multipart, in which case it's the hash of the hash, oh yeah.... So you should split it in advance if you want to verify the hash (more fine print).
Worked great. Good for you for sharing this project, very cool.
I can confirm this, to some degree.
We have larger customers with 20 or 40 or 80 TB of data to bring into rsync.net and everyone is always very interested in physical delivery, which we offer, but it's always easier to nurse along a 20-30 day transfer than ship JBODs around.
As long as you have a transfer mechanism that can be resumed efficiently (such as zfs send) and you don't have terribly bad bandwidth, we always counsel to just run the very long transfer. It does help that we are inside he.net in one location and two hops away from their core in two other locations and we can just order 10gb circuits on a days notice ... because he.net rocks.
E.g. compare a soap bubble versus a bubble gum bubble. It's a lot easier to scale up the soap bubble, which is not elastic.
Moreover it turns out that elasticity is a very valuable quality of a cluster for most workloads; we want this intuition to be true, that our cluster meets resistance as it grows, in the sense that it will shrink when the workload decreases. This matches our economic intuition, too. We want this so much we have to build another software layer to make this happen - e.g., k8s.
During the transfer, the AWS sync constantly failed due to random issues which drove up the total time to transfer the files. Something like having a tilde (~) in the filename will totally break the sync. You really need to keep track of where it failed. We were constantly trying to craft additional rules into our sync logic to catch the 'gotchas'.
Another point you alluded to was the Etag/MD5sum that's stored in AWS. Pretty useful if you know how to use it...