How an empty S3 bucket can make your AWS bill explode | Hacker News Reader

How an empty S3 bucket can make your AWS bill explode

medium.com

311 pointsmacok111 comments

Looking at AWS's price page for us-east-1 [1]

* PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests

* GET, SELECT, and all other requests are $0.0004 per 1,000 requests

It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be unable meet durability goals. Successful read operations are an order of magnitude cheaper because they likely move only one of several heads, sometimes none at all (because they're cacheable).

Authorization failures should never hit an HDD.

Other folks have mentioned that these auth failures are costing AWS something. Sure...but very very little. Far less than the cost of a successful GET, much less a successful PUT. They really shouldn't be charging at all for this, or if they do, it should be like $0.00001 per 1,000 requests (another order of magnitude cheaper than GETs). Not even that really; the actual cost for them to process this request is likely less than they're (over)charging for the bandwidth alone.

[1] https://aws.amazon.com/s3/pricing/

What's crazier is that turning on Requester Pays does not actually mean requester pays when the request is a 403 [1][2].

Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill...

That's a platform level security issue for AWS to resolve. We do not need another Bucketgate.

[1]: https://twitter.com/Lauramaywendel/status/178507498639629151...

[2]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...

For me it looks like AWS has mainly different scenarios to handle. First, unauthenticated users are a big security pain for AWS and S3 because S3 is used for everything. But S3 was not designed for "secure" public access. As a result billing attacks are possible (e.g. [1]) and will not be fixed (by design). Second, authenticated AWS users are kind of trustworthy and these are never assumed to do any DDoS stuff or similar. Here, we have even more surface for billing attacks. I am waiting for the moment when this assumption breaks because credit cards do not make users trustworthy.

My personal suggestion is that you should never use S3 for public access. Public buckets are an open gate to your AWS bill. If you have private buckets the attack surface is reduced but even in this situation you cannot prevent billing attacks. Try to reduce the damage by activating Cost Anomaly Detection.

[1] https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-a... [2] https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-...

private in the sense that the connection will never make it to the S3 bucket, is that correct?

private in the sense that it is locked down to allowed source IP's or IAM-- then the connection still gets there and AWS incurs a charge for the Access Denied response, etc

Yes, private in the sense of "block public access" (your second point) (https://docs.aws.amazon.com/AmazonS3/latest/userguide/access...)

Sadly, the private in the sense of "never make it to the S3 bucket" is nearly impossible. May this can be achieved by choosing some "secret" S3 bucket name.

> I am waiting for the moment when this assumption breaks because credit cards do not make users trustworthy.

IMO it is already broken, or beginning to crack at least, with the advent of throwaway virtual cards.

> Try to reduce the damage by activating Cost Anomaly Detection.

Yeah, this sounds like it could be a better solution than just using CloudWatch Billing alarms and billing alerts that I mentioned elsewhere. It's hard to tell at-a-glance if Cost Anomaly Detection is built on top of CloudWatch Billing alarms, or if they're just totally separate products. I don't seed the latter mentioned in the docs for the former.

[1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori....

What is bucketgate? Your comment comes up when searching...

I'm referring to the endless amounts of S3 buckets that ended up inadvertently exposing sensitive data publicly due to how complicated and confusing the various access control methods were / are. And also, Amazon very eventually doing something about, probably a decade after it was needed. [1][2][3]

[1]: https://cloudsecurityalliance.org/blog/2023/04/06/the-data-o...

[2]: https://news.ycombinator.com/item?id=18468753

[3]: https://news.ycombinator.com/item?id=34585670

That's why AWS doesn't allow pentesting of this (S3) service.

> Other folks have mentioned that these auth failures are costing AWS something.

This should be entirely irrelevant, as those same failures were happening even before the bucket existed, they were just hitting a different failure path.

However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs--

Aaaah! CloudTrail data events are priced in such a way that each S3 request gets logged--at a cost of 20x that of the S3 request itself. I really hope OP got THAT part of the bill waived as well.

Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.

My first thought is that can be abused to multiply the damage if you want to engage in some cost based denial of service.

About every other week AWS gets out of their way to make us remember that they are the worst cloud around.

For reference https://github.com/ZJONSSON/node-unzipper/issues/308

To preface this comment, I'm not super familiar with range GETs or the library you're using. Are you saying that the library was performing range GET requests but instead of sending a byte range, it was only sending a starting byte number?

> Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.

Re the bill itself - You may be able to get the surprise bill waived by AWS with a support request, especially if you haven't had to do that before.

Re the metered billing - This isn't super clear anywhere I could find in the AWS docs, but I think from AWS's perspective your network data transfer out from S3 is the amount of data you requested + initiated, not what your app ultimately happened to consume from the stream. They really do not make this clear though, which might help your case.

FWIW, it sounds like AWS's implementation is behaving consistently about "range specifiers" with the RFC they link to in the S3 docs.

https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.1-4

Sidebar, but: If it works for your setup, I think putting a CloudFront distribution in front of the bucket and accessing that way would mitigate the damage by caching the whole object after the first request. I've used a setup like this for streaming very large files to many viewers before.

https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

Yes. It made many request with Range star- (i.e. without end).

500$ is not making a huge dent in the monthly bill for this. I just got a few laughs at my expense from colleagues..

I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then. I could generate 10s of gbps of "apparent" billable traffic with a puny VPS.

You could also generate 10s of gbps of actual billable traffic with a puny VPS by forging TCP ACKs and getting AWS to send you data faster than you can actually receive it, but that's closer to an actual DoS attack and more likely to get you in trouble...

> I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then.

I suppose that would be easy enough to test.

I was curious how video streaming services handle the case where a stream is started or partially consumed. Mux meters it in terms of [seconds of] video segments delivered:

> Cost is per minute of video delivered. To calculate video delivered, we measure the number of seconds of video delivered to a video player. Note that if a segment of video is delivered, it is charged, even if the viewer doesn’t actually watch the video.

[1]: https://docs.mux.com/pricing/video#delivery

I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?

This is exactly where my mind went. I assume most people, myself included, didn't realize this. This is absolutely going to turn into the economic equivalent of a ddos.

I'm far too lazy to do the math, but I do wonder how many typical residential connections it would take to make a noticeable dent on a big company's bottom line with this method? I'd wager it's a surprisingly low number.

A bored or malicious actor can also just DDoS your CloudFront endpoint and run up an effectively unlimited bill on your end. That exposure is par for the course with AWS, they don't have a spending hard-stop and at this point they probably never will, you just have to hope that they have mercy and waive the bill if you get a surprise extra few zeroes on the end.

and their bills cost up to 100x what they should (and then other services that run on top of AWS may inflate it 10x again)

Shield Advanced claims cost protection, although as always, the devil is in the details... - https://aws.amazon.com/shield/features/

"AWS Shield Advanced comes with DDoS cost protection to safeguard against scaling charges resulting from DDoS-related usage spikes on protected EC2, ELB, CloudFront, Global Accelerator, and Route 53 resources. If any of these protected resources scale up in response to a DDoS attack, you can request Shield Advanced service credits through your regular AWS Support channel."

Shield Advanced is $3000 per month with a 1 year commitment, plus extra bandwidth fees on top of your existing bandwidth fees, I somehow doubt that many of the users who would be impacted the most by an unexpectedly large bill (individuals and smaller businesses) are proactively paying that much for DDoS insurance just in case.

Agree on that. Was not claiming that Shield Advanced is the solution for this nutty request Rodeo...or maybe we should call it 404 festival...

I think CloudWatch billing alarms [1] are the only viable "solution" for an average joe here. Of course, they do not actually stop the usage, but at least they bring automated awareness.

Even with that, there's no easy way to rename an S3 bucket [2], plus doing so would commonly require updating config + re-deploying any apps using the bucket too...

All of this to say, the customer level really feels like the wrong layer of abstraction to solve this problem.

[1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

[2]: https://stackoverflow.com/questions/41511034/how-to-rename-a...

Press F12 to screw competition

It's not just "kind of insane", it's completely bananas. Like what the actual fuck?

We really need some kind of regulation to ban predatory cloud pricing.

This kinda happened with Netlify recently (https://www.reddit.com/r/webdev/comments/1b14bty/netlify_jus..., https://news.ycombinator.com/item?id=39520776). They got a bill over $100,000 via spam requests.

Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own.

One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.

> Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you.

Who pays when the unauthorized requests contains truly random data, i.e. they point at an org and bucket that do not exist? Amazon does. Part of their cost of doing business is building the infrastructure that routes authorized requests correctly, and if a request is for invalid data, it's dropped. That should be par the course.

Large numbers of unauthorized requests can be effectively considered a DDOS on the service itself. That the attacker can add a note onto their attack that says "btw bill X customer for all this" and have that honored is ridiculous.

> They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you.

They cost a lot less than a multiple disk write (PUT), a disk read (GET).

They cost the same to AWS whether the target bucket belongs to me, Amazon themselves, or does not exist.

And in this case, the customer definitely didn't "screw things up".

Charging for unauthorized requests is just weird and customer-hostile. Like, how am I supposed to prevent that?

I get that a service provider might want to "punish" a customer who misconfigures their systems and DoSes themselves with e.g. no backoff on 403 error, but I find it hard to believe that that's the common case -- and how is typo in authentication supposed to be so different from typo in bucket name?

That's why, even though during the day I work on AWS for people who pay me to do so, in the evening I work on my startups hosted on Hetzner where I know in advance what my bill will be not only this month but also this year.

Same here, it might be slightly less convenient but boy how well I sleep at night.

First of all, I agree this is all bananas:-) but it seems is the same with GCP and Azure. For Backblaze B2 it seems its unclear in the documentation. Willing to be corrected on this....

I'm less familiar with AWS. But on Azure, you can have private storage accounts which aren't internet routable so they wouldn't even be able to hit the endpoints to rack up charges. It's a bit ridiculous that you'd have to go that far to avoid being charged for unauthorized access, but I'd be surprised if AWS and GCP didn't have similar capabilities.

I don't know if Backblaze charges for failed requests, but it lets you set spending limits, so the worst case is that the attacker drains the daily quota that you set and later requests fail (DoS), not that you get a surprise $1000 bill.

Not in GCP if https://cloud.google.com/storage/pricing is to be believed:

> Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.

Heh, charging for 5xx would be even more insane than charging for 403. "Our system is not working correctly at this moment. That'll be $0.25."

>kind of insane that unauthorized requests contribute to your bill

Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly

That's exactly it. I just spent the morning cleaning up my buckets.

I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private.

Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already.

So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances.

Obviously all accounts should have budget alerts, but there is no budget ceiling feature.

Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.

You cannot shut down an S3 bucket with a lambda tho, the mentioned issue above is only solvable by deleting the bucket.

Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people.

Ok I'm actually going to delete my web site and cloudfront distribution over this. It gets no traffic (at the moment).

The bucket is the same name as the domain so I can imagine this could be exploited quietly very easily as PUTs are so expensive.

Literally a $5 VPS could cost someone $1k.

i doubt it. i would imagine any single ip address would get rate-limited/throttle before doing much. the situation the article is describing is much more like a botnet — there were hundreds or thousands (or more?) of unique instances making requests.

I very run big buckets in production that have single addresses hammering them hard all day every day. I've never seen any restrictions or throttling.

> i would imagine any single ip address would get rate-limited/throttle before doing much

Yes, you would imagine, because without relevant WAF rules in place there would be no throttling.

> i would imagine any single ip address would get rate-limited/throttle before doing much.

Does AWS have default per-IP rate limits?

A $5 VPS should have a $10 spend limit or something, at which point it should shut down. In a reasonable world.

I think they meant the other way around, someone pays $5 for a VPS to spam unauthorized requests to your S3 bucket to get Amazon to bill you $1000.

Thanks. Damn, I've misunderstood some things recently so I'll take some rest and slow down I think.

Well this was an unwelcome attack vector, since I wrangle a fair few S3 buckets. I hastily threw together an S3 bucket name generator in Bash under Linux [1] to create names that follow the AWS rules to start my migration to my new S3 bucket names, and welcome any help to ensure it creates names that are strongly resistant to this attack, and follow various S3-service-like vendor rules, so I can do what I can until AWS and others come out with a stronger defense.

[1] https://github.com/automaticit-anthonyyen1/s3-bucket-name-ge...

Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people. And if you are using static websites, then amazon's own docs require you to use the website name as bucket name.

Quite terrifying. Anyone know if competitors also bill requests which result in a "permission denied"? GCP, Azure, Backblaze B2, etc.?

It seems nobody replied to you, so let me try.

Yes for GCP and Azure, also for GPT...For Backblaze I am still investigating...

Two quick examples in different contexts:

https://learn.microsoft.com/en-us/answers/questions/1064419/...

https://community.openai.com/t/cost-for-failed-gpt-requests/...

Edit: It seems for Backblaze that would be a no, see below:

https://www.reddit.com/r/backblaze/comments/16o4bed/charge_b...

Gcp def bills requests but not sure about unauthorized requests - this needs testing

90% sure GCP charges for rate limited API gateway. i.e. doesn't matter if it is above rate limit. Never got 100% confirmation either way though - their documentation on pricing is too vague

Yea they do, it says always charge when listing buckets.

Any way to mitigate this on AWS by restricting access to it via VPC only?

This note at https://cloud.google.com/storage/pricing suggests otherwise:

Crazy, I wonder what AWS's margin is on these unauthorized requests? Questionable incentives.

Guess what? It seems is the same with GCP and Azure. I am willing to be corrected if it's not...

Yes? AIUI all 3 are regularly accused of wildly overcharging on egress; what's your point?

My point is that we need to talk about all three.

could you solve this with a "requester pays" bucket?

not viable for a public bucket, but seems like it'd do the trick on private buckets.

edit: hahaha, i'm wrong https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...

There are so many things that can make your AWS bill explode, so it probably should be included into your budget. Like, $1000/mo is not actually $60K in 10 years, but $60K + some random $20K that may or may not be waived by AWS support.

AWS shouldn't allow bucket identifier reuse if they're going to charge for unauthorized access.

Eye opening; I deleted my (empty) S3 buckets in response to this.

uf, aws should definitly not count the requests against a bucket that is not configured for public use.

Edit: if they do that, they should make it possible to firewall buckets via ips so that they are only accessible via certain ips

Or at the very least the requester should b charged for the failed auth.

How is that possible? AWS can't bill anyone on the planet.

This is an unauthorized S3 API request. If there is a valid access id and secret attached, bill them for the request.

I doubt that anyone's agreed to pay that. Also that there's often a valid access ID and secret. Also that the cost of scanning through all the client IDs across all secrets is worth the money.

already a meme https://i.imgflip.com/8ohpwh.jpg

I wonder why does not Amazon be responsible and lock these accounts clearly attempting abuse? Logically either they are compromised or malicious. At least after certain number of failed request. And thus should be treated as such.

Anybody can do this.

This case is especially problematic because this person was being hit from many distinct and legitimate accountants.

One problem here is that you can execute these unauthorized S3 requests without having an AWS account at all.

For me, this is not only a cost thing, but a security issue as well. Considering the IAM auth or any kind of auth method, the goal is never to be "impossible to break-in", but "the spending of break-in is higher than the potential value it protected". Yet if the price of break-in is paid by the victim, this will be a whole different story. Someone can spam the S3 API with brute force IAM credentials can try out everything until find the right credentials, and the failure attempts are paid by the victim, which is insane.

Great find. This has serious exploit potential.

Crazy. Would not have expected that.

Unfortunately you can't always keep your bucket name a secret, if you use presigned URLs to let your users directly download objects...

It’s a good idea to use a reverse proxy like Nginx to handle the transfer through an internal redirect, which will hide the bucket name and authentication credentials.

But this means data transiting your network instead of being directly fetched from AWS infrastructure, which for me mostly defeats the purpose of presigned S3 URLs.

If your server is on AWS, then the egress cost should be identical whether the transfer originates from S3 or EC2. But if your server is on premise and your host charges for egress (or has impactful bandwidth restrictions), I can understand where you're coming from, as you could end up paying twice.

Introducing complexity, failure point, scalability issue just to work around broken billing. You are probably right but I feel it’s wrong architecturally. I always liked S3 precisely because I could offload file serving to them.

Use a password generator to create your S3 bucket names.

This doesn't solve the problem much.

1. If you are using buckets for static website hosting, amazon's own docs require you to use the website name as bucket name.

2. If you are using pre-signed urls, they expose the bucket name.

3. Bucket names must be between 3 (min) and 63 (max) characters long and have various limitations. So, it's not too hard to guess bucket names and run millions of requests without even having an account.

Since you need to create a unique bucket name any way, I like to include my AWS Account ID in my bucket names. Be advised that bucket names need to be 63 characters or less. Another reason I do this is that CloudFormation supports the use of:

${AWS::AccountId}

This does not save you because AWS Account IDs are not private, e.g. if you are using S3 pre-signed URLs anywhere, those URLs include your account ID as part of the X-AMZ-Credential URL field:

https://medium.com/@TalBeerySec/a-short-note-on-aws-key-id-f...

Lesson 4: mediocrity has become the most effective path to fame in a modern (and mediocre) world.

What's next? An article that explains why the bill stacks up when you log all denied fw packets?

Many are speculating that this will not be fixed by AWS (by design) however now that this has been discovered AWS will "need" to repair this flaw or they will start incurring customer flight to more secure or cheaper services.

The question is more about how long AWS are going to take to fix this issue and how many DDoS bills will they forgive.

https://news.ycombinator.com/item?id=40221108

For a long time, bucket name is believed to be non-sensitive so they are sometimes placed in public just for convenience (like in CI scripts of public Git repositories). It's easy to collect examples by searching "s3://" on sourcegraph. I believe it's much easier to find a bucket charging a company/open source project/person than we expect.

I was in the same situation as OP a few months back. My bill was $3000 after creating a "vanity bucket" which saw 500M accessed denied POST calls over a month. I took a while to get a one time refund.

This isn't obviously a scam to me. Cloudflare have guarantees around pricing per request, and I'd expect them to honour it. AWS have different prices. I'd expect them to honour them.

The problem is the cloud priorities availability over cost savings. We used to run our own infrastructure, that would just fall over if it got too many requests. This infrastructure never falls over; it just bills more. I don't know the right way to solve this, other than don't stick your S3 bucket on the public internet.

The right way is for unauthorized requests to not cost anything.

That makes no sense, though. Some network and compute are running for this to happen. Someone needs to pay the bill. It's not obvious it should be the cloud service, if that's not the agreement.

This is part of doing business for the cloud provider.

The cost of running compute to deal with unauthorized requests for an arbitrary extant S3 bucket, is the same as the cost of running compute to deal with unauthorized requests for nonexistent S3 buckets.

If I generated a billion requests to an S3 bucket that did not exist, Amazon would have to decline that traffic in the same way. Since the recipient did not exist, there would be no one to bill.

I as the attacker should not be able to add on a sticky note saying "btw you can charge X user for this malicious traffic" and have Amazon actually honor that.

EDIT: Here's an analogy. Say a business in a city has a front lawn that they must pay to maintain. Sometimes people walk on the grass as a shortcut to the business. Some subset of those people also enter the business as a customer. Would it be fair for the business to charge those people more to account for the extra landscaping bill they cause?

But the answer doesn't matter, because that's not what happened. What happened here is more like, someone left a note on the front door of the business saying "My name is X, and I walked on your grass last night", and so that day the business charges more to any customer whose credit card says their name is X.

They charge ten times as much for an unauthorized PUT as they do for an unauthorized GET, and I feel fairly confident in saying that it does not cost them ten times as much.

That seems like a different issue to "who should pay at all".

When thy receive a request to a non-existent bucket why don't they charge it to the lexicographically closest customer?

AWS rakes in multi billion dollars in profit every year. They can definitely eat the costs of unauthorized requests to s3 buckets.

At this point I believe that convoluted pricing is their business model. I got hit by their “tiny font price” twice, paying extraordinary money for trivial things. And I barely used AWS.

Are Lightsail buckets which have predictable prices at risk too?

If there are limits which are surpassed, S3 charges may be incurred.

Just waiting for a botnet group to abuse this, buy puts on the expected shortfall in earnings and profit.

Why would there be a shortfall in earnings? Failed (due to being unauthorized) HTTP requests cost AWS ~nothing.

A quick answer is to put the bucket behind cloudfront so that the denial is cached, but that's not totally free either.

If they actually charge for unauth'd requests that's absurd

> Other than deleting the bucket, there’s nothing you can do to prevent it. You can’t protect your bucket with services like CloudFront or WAF when it’s being accessed directly through the S3 API.

You can't protect your bucket with CloudFront if it's being access using S3 api.

That doesn't work, the standard endpoint always remains available.

I assume random names are the only way forward. Unfortunately a bucket name can’t be longer than 63 characters.

Surprising people are not already doing this. Something that has been beat into me for at least a decade.

You can stuff >256 bits into 56 BASE32 digits. The 63 character limit seems like it would only be constraining if humans want meaningful bucket names.

Reply on news.ycombinator.com