Hacker News Reader

Comment by kondro | Hacker News Reader

I’m surprised none of the engineers either knew about, or thought to ask AWS support, S3 Batch Operations or S3 Replication.

Both of which could’ve completed this process in a relatively short amount of time in a completely hands-off way.

When I Google for “move large S3 bucket” the first result I get is an AWS article which goes through all the best options: https://aws.amazon.com/premiumsupport/knowledge-center/s3-la...

View on HN

> I’m surprised none of the engineers either knew about

You'll probably also be surprised to learn that the original article not only specified that link but also that they chose the first listed option, which probably wasn't a surprising choice, given that it was listed first.

You’re right. I was tired when I first read this last night on Reddit and I could swear the link wasn’t there then.

I'd bet good money that this team was tired, when they read the docs/googled as well.

If you read the thread, they mention that they had a requirement to empty the original bucket within the maintenance window, as well.

Which you could’ve easily done with S3 Batch Operations or the EMR-based sync in a super-parallel way and little engineering effort.

Their biggest issue was letting someone who didn't know a thing about S3 dictate the rules. Revoke all access to bucket ~= everything is deleted.

Then they could set a lifecycle rule, wait a day, and it would be gone for real.

From my experience if you have a bucket with billions of objects it will still take a few days after the objects are deleted by the lifecycle rules to be able to delete the bucket. The objects are non-recoverable but I believe due to the eventual consistency nature of S3 you can't delete the bucket until the objects are all fully deleted.

Yeah, S3 is frankly a disaster. People use it like a filesystem and if you have billions of objects, it very quickly becomes a nightmare to manage. Size of data is a complete non-issue, it's all about the number of objects.

I will point out deleting that much data from harddrives isn't a fast operation either if you were storing it on drive. Same for tape.

Sure you can destructively destroy the drives. But you could also just use KMS on the S3 bucket object and then toss the key and close the buckets. Gonezo.

except you'd still be on the hook for the monthly cost for the useless bucket full of data

> I will point out deleting that much data from harddrives isn't a fast operation either

Dropping a btrfs subvolume or snapshot is pretty fast. ditto for zfs.

That doesn't help if you want the data to be gone instead of hidden.

If it's encrypted then destroying the key gives you a blank-with-noise drive. There's no equivalent for S3.

It's strange that people just shrug their shoulders and accept this state of things in 2020.

Any system these days should allow near-instantaneous admin operations for just about any reasonable request. Subtree move, delete, or rename should all be instant. Similarly, any simple ACL changes should be instant.

Similarly, data movement should scale linearly with the volume of data, not the number of objects.

I'm definitely one of those who treat it like a filesystem :') . I'd love to hear a different point of view. Would you feel ok to share thoughts on how I could be thinking of s3 to fully appreciate the right way of using it?

That entirely depends on what you're using it for. Mimicking a typical block-level filesystem with object storage will lead to problems beyond a few thousand objects or shallow "folder" levels.

Object storage is best for larger files with bulk updates. Either write new files entirely (best for S3) or append/replace where supported (like Azure/GCP). Any further structure and metadata should be maintained at a higher level with pointers to the raw files in the object store.

There are products like Stablebit Cloud Drive [1], ObjectiveFS [2], and other storage companies that handle all this for you. You can also look UtahFS [3] to see open-source code on building something yourself.

1) https://stablebit.com/CloudDrive 2) https://objectivefs.com/ 3) https://blog.cloudflare.com/utahfs/

> wait a day

How does that solve the problem of having the bucket accessible and empty in two hours?

I had to do this before batch, replication, inventory. Support, SA, TAM were all like... good luck, though they did offer to make an inventory but they delivered that two weeks later. Everything at the time said s3distcp was the only option. It honestly wasn’t that bad, except it did a serial list before it started copying which would have taken days so needed some tomfoolery.

At least when I look into bucket replication (a long time ago, I don't think it was GA at the time), the sync operation only affected _new_ keys, not existing keys, so it wouldn't have worked for this purpose.

By default, you'd have to do something to touch the keys. But AWS support is more than happy to fire off a full init for a new replication if you ask them.

Reply on news.ycombinator.com