Amazon S3 now allows multi-object delete
aws.typepad.com
aws.typepad.com
So, I use S3. A lot. Only, I don't store many bytes: I just store lots of objects... enough that I have in the past estimated that I owned >1% of all objects on S3.
(http://news.ycombinator.com/item?id=2154792 <- here I describe how I had been making a billion PUTs per month when Amazon comes out and claims they only have <300 billion objects stored in S3.)
However, in the last few days? I've binged, and this week I'm purging. I've been doing as many as 10 million (and never less than 4 million) deletes per hour for /days/.
Why? One of the things I use S3 for (storing SHSH data, a giant man-in-the-middle attack against Apple's dysfunctional iPhone software protection systems) was simply wasting space. :(
(Specifically, I realized that I had, due to some debugging code I never really realized was for debugging only and thereby never got around to deleting, had been storing every single request in and out to the service throughout its history in XML form, in addition to the tightly encoded binary.)
The thing to realize here: DELETE requests are free. PUT/LIST requests are $0.01/1k, GET/HEAD are $0.01/10k, but DELETE requests are not billed by Amazon at all; even the bandwidth for them "won't count" if they are made from EC2 (which I'm doing).
However, one can only imagine that deleting a billion objects in a few days with a billion separate tiny requests to Amazon is causing someone, somewhere, a bad week (and possibly even a ton of meetings). :(
(Which, honestly, I feel sorry about, but my business isn't even that profitable, so I've been having to figure out ways to cut costs, and this useless data is apparently costing me nearly $1k a month.)
Well, with all of this in mind, and now after almost a week of "the onslaught" (and even myself wondering multiple times what this many requests must be doing to their stats), Amazon conveniently announces a new API feature that lets me decrease the number of separate requests I'm making to their servers by a thousand times over...
...really, I cannot help but wonder ;P.
Seriously, though, this new multi-delete is EPIC ;P. I am now deleting objects at an rate of over 4 million per minute! (Although, as these are POSTs, I wonder if these now cost me money; that said, it likely isn't going to much.)
(This is fast enough that I'm now "cleaning house" much more thoroughly than before: going through some of those "database buckets" I have and deleting obsolete indexes.)