Hacker News Reader

Comment by booleanbetrayal | Hacker News Reader

TL;DR - The delete is the hard part. You can use S3 batch operations with an inventory list to do the copy very quickly. Alternatively, you can setup replication and "touch" each file using a self-copy CLI command, once the replication policy is in place. For the deletion, you're sort of stuck with lifecycle policies, which would take a day or so to clear out the old bucket, but could be supplemented with manual interaction, I'd imagine. There probably needs to be a better mechanic for completely wiping buckets. Last I checked there was not.

View on HN

Yep. Had to delete a bucket with 100 million or so objects in it a while ago, with each having multiple versions. So it could have been a billion objects. It was something I needed to run every now and then (process to clear the production bucket while getting ready for the full cutover) and have it done in a few hours rather than wait for lifecycle policies to kick in.

I ended up hacking together this https://github.com/boyter/aws-s3-bucket-purger and then ran it on a few different machines to clear it out in under an hour.

Not having some easy way to clear buckets was very annoying.

S3 Batch Operations can also invoke a Lambda function per object, so it's straightforward enough to create a function that calls DeleteObject() and let Lambda scale out to silly levels.

encrypt all objects. Wipe bucket? just toss key and delete bucket, done.

Use generated bucket names not fancy bucket names, buckets are cows not pets.

Only, that's not actually deletion, nor would it have helped their business requirement:

> It is a 3rd party application that puts data into that origin bucket. They needed the bucket to be empty before the new version gets activated. And they wouldn't use another bucket. Something out of our control

s3 just makes you work for it:

- delete 1000 objects per post to /?delete

- do this 100 times per connection

- do this on a few threads per server to avoid being throttled

- which entails keeping track of which servers you're using (there are hundreds of servers available but dns only exposes a couple per second)

..and that assumes you've already got a list of keys - if you need to also enumerate the bucket you'll have to employ the same strategy and also partition your queries somehow to get multiple non-overlapping continuation-tokens for your concurrent calls to /?list. you can also request a bucket inventory but if you're in a hurry it's faster to do it yourself (for a 25TB bucket the turnaround on the inventory request is probably 2+ days)

> The delete is the hard part.

Correct.

> you're sort of stuck with lifecycle policies, which would take a day or so to clear out the old bucket

I've had delete jobs run for a month per bucket. We're talking terabytes per bucket, but even so.

Reply on news.ycombinator.com