Migrating 23TB from S3 to B2 in 7 hours

Ain't aws great would've being like what $250 on DO

12 bay Synology is $1,600, load it with 24 4TB drives for another $2,000

This is absolutely sneaky and would work. I believe S3 also has file access cost but it's minimal.

If you had a partitioned list of what you needed to move per node, so that coordination was minimal between them, this would be pretty viable.

LightSail has (or at least had) a hard limit of 20 instances. They also have a soft limit of 2 instances, after which you must request an upgrade to a higher limit. I had to submit a support request explaining my intended usage. It took a week to get approved.

The stated reason for these limits is to avoid unexpectedly large bills. But I suspect that it's also to prevent crazy-ass strategies for getting around bandwidth costs.

Lightsail instances have terrible bandwidth throughput. It has high limits because it's all overbooked low-priority traffic shunted off their network as soon as possible.

I evaluated moving 2PB with snowball v. putting in 10g/100g links. The issue with snowball ( i started a company that did what snowball did and shut it down (failed )) and other fedex/RAID solutions is that you have 3 transfers. You think the LAN transfer will be quick - but you're generally rate-limited by the systems more than you are by the bandwidth-delay product. If you're in a high traffic DC area, it's pretty easy to get temp bw or install circuits to carry that. 10g for 2pB is 18 days of transfer - which sounds like a lot - but that's 5 days of transfer on each site, 1 day of setup, and 1 week of shipping. Those numbers aren't real but they're close.

So, snowball works in a lot of area, but like so many AWS products, it works if you adapt to it.

pigz/scp/zstd works extremely fast in line.

In your case you're pulling from S3 to another object store.

I moved ~1PB from one S3 region to another. "Why not use replication," they asked. That only works if it's turned on when you upload the object - another fine-print 'gotcha' in the easy AWS service. Then you get into rate-limits. In 2010 I asked AWS if I could spin up 1000 servers to test something - nope - elastiticy at that level is for the big boys.

Now I work for a large cloud company and we still run into elasticity.

To move the 1PB from one S3 region to another we spun up hundreds of spot instances (oh, we were compressing and glacierizing it too) and built a perl/mysql batch job "s3 get | zstd | s3 put" process and parallelized it. One thing nice about S3 is it pulls the md5 hash - unless multipart, in which case it's the hash of the hash, oh yeah.... So you should split it in advance if you want to verify the hash (more fine print).

Worked great. Good for you for sharing this project, very cool.

We have about 125TB on Wasabi. Costs about 650/mo. Their performance for large files is great.

I also recommend Wasabi. Even cheaper than B2 and they follow the S3 API so all existing tools still work.

They also have different billing models with a free egress plan if you aren't downloading data that often.

It's built for off-site backup type workloads, where you write once and read hopefully never.

You are probably using it wrong.

I was pleasantly surprised to see that the previous rather crazy retrieval pricing model (price based on the peak retrieval bandwidth you used at any time in the current month) was replaced by a straight price-per-GB model a couple years back. That made my life a lot easier when pricing up Glacier vs B2 for a client.

If it's that much churn, wouldn't it make sense to point new writes/updates to B2 and let s3 naturally drain/lifecycle out its data?

Sorry for the bad vibe.

I was trying to interject some humor into a dry process. When I asked a couple peers from my storage background, "how would you do this project" they both said the same thing: 1) form a working group 2) do a study 3) test 4) 90 days later, do an analysis. Epic process.

We care a lot, but that's beyond our scope. This was an all-hands reviewed process, and like machines that are built to be ultimately reliable including: hospital generators, human-rated spacecraft equipment, airplane engines for single-engine remote location flying (PT6), our process was detuned speed-wise specifically, for overall service reliability and minimal impact.

We tested, we analyzed, we saw weird failures (python core dump?) in simple code, and followed up on all of those before proceeding. We agonized.

In the end we were satisfied, because our customers couldn't tell anything changed, and our CSR's got zero new tickets. Not bragging at all, relieved of course, but, we care about the CSR's load and work experience too.

Thanks for checking the article out, and let me know if we can show you more about our service.

We do use Cloudflare, but a lot of the instance backups we store are multiple GBs - Cloudflare doesn't cache those. Not to mention uploads from regions like Singapore can be very slow all the way to the US.

It's not a deal-breaker for us, but we're very much looking forward to when they can support more regions.