all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!).
Try to beat that reliability guarantees with whatever you handrolled, and I bet you will never be able to beat the cost of S3, even match the durability, reliability, availability guarantees at any reasonable cost at all
from https://aws.amazon.com/s3/storage-classes/:
Key Features:
Low latency and high throughput performance
Designed for durability of 99.999999999% of objects across multiple Availability Zones
have you ever built anything with 11 nines? (as in eleven nines)S3 API support sounds great until your costumer builds a system with an "S3 compatible object storage" product. Soon you discover that many "S3 compatible" solutions aren't actually that compatible when pushed.
Then it costs $92 / TB to get it out again.
Also S3 has durability guarantees but it's very difficult to do a durable transactional write to S3. Try it a few million times and see. The API is a defacto shitty standard.
These two facts are rather interesting when it comes to doing a restore from your supposed backup or wonder why consistency guarantees between external metadata services (DB) and what is in S3 don't always line up.
if it is for migration: it is one time cost that anyone can swallow easily if they decided to leave AWS for something else.
If your data is worth < $94/tb - it is really not worth pulling it out of AWS. Just let it sit there.
or just use cloudfront to download your data ($8.5/Tb)?
this checks out.