Hacker News Reader

Comment by morningsam | Hacker News Reader

>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does.

If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for someone from their PR department to see it. Only then will they finally waive the costs.

Good on Fly.io for trying to handle such situations more sensibly.

View on HN

There's plenty of situations you don't hear about.

A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.

You likely already know that, but to anyone else interested: a good way to prevent these kinds of situations is to run 'nosey parker' on your git repo before pushing it to a remote. It will dig through your code and configs, looking at files and through all the git history, and highlight anything that looks like tokens, passwords, keys, etc. You can set it as a pre-commit hook to block the offending code from even being committed.

https://github.com/praetorian-inc/noseyparker

https://github.com/trufflesecurity/trufflehog is a similar tool but checks for far more secrets, so I think it'd be a better choice.

Github has a similar feature that’s free for public repositories IIRC.

I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?

It's been a few years, so I'm going off of memory, but it was mostly best practices stuff (enabling Cloudtrail, rotating older keys, etc). Anything to ensure that once the attackers no longer had access, removing/monitoring anything that would have longer term implications.

this makes sense; plug the hole and stop taking on water before you ask for a free cleanup.

> If the incidents that made the rounds here in the last few months are any indication

They really aren't. There's a huge world out there beyond the HN front page.

What are you referring to, exactly? That doesn’t sound like AWS at all.

AWS are very well-known for bill forgiveness. It’s not something set in stone, but if your bill explodes accidentally, even due to a mistake you made, they will normally forgive it if you ask them. You don’t need to go running to social media at all.

You're right, I was getting two relatively recent posts mixed up:

1. After a DDoS attack, someone got a $100k bill from Netlify for his static site and after he asked to have it waived, they generously reduced it to $5k. Only after his posts about it blew up did Netlify waive it completely. [1]

2. Someone got a $1k bill from AWS because lots of people made _unauthorized_ requests against his empty S3 bucket. AWS did agree to waive it immediately, prior to any social media posts. [2]

I probably just remembered (2) as "that ridiculous billing situation involving AWS" but got the details of what exactly happened mixed up with (1).

[1] https://news.ycombinator.com/item?id=39520776

[2] https://news.ycombinator.com/item?id=40203126

For (2), AWS _also_ has now shipped a change that makes those requests non-billable permanently: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...

I have personal knowledge of two cases of this happening and both were immediately dropped. One was a high school student's personal project gone awry for like $3k worth the other was a startup where it was $120k or so. Two different providers. Neither had to particularly argue their case, just ask and wait a few days.

We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.)

So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.

FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...

Thanks, we are trying to compose an argument with this service in mind, we didn't have any recursive invocation protections for this lambda, and the AWS services it hammered during its invocations contributed to the massive cost.

I was a PM in GCP and refund requests due to customer misconfigurations would make it to me for approval.

Generally, I tried to make exceptions to grant the refund. It sucks to get these bills and they can be quite scary.

Unless you have a real chance of bursty traffic I suggest going with compute that has more predictable costs.

So no elastic anything. Just servers?

If you have elasticity then make sure you understand the limits. Even Serverless has these knobs but for various reasons they default to high elasticity.

But many folks don’t need the elasticity at all. So you should factor that into your architectural decisions.

Well, the ones where they forgive it right away don't write blog posts about it, so you don't know about them. I guarantee that every case which makes the front page is either an example of a mistake made by an individual support engineer following the wrong script, or the person posting it is being dishonest about what actually happened (I recall a particularly suspicious recent one where they hid the name of their site). The cloud providers all have a policy of forgiving these sort of cases. The only speedbump is ascertaining if it was a genuine mistake.

Another reason to be cloud agnostic. They are counting that switching cost will make you eat the bill.

If you have to be cloud, do dev in one cloud and test/prod in another.

I know, I know, easier said than done.

An argument that is often only made by folks who didn't ran anything at scale on a public cloud. If you've ever set in a room when a cloud deal was signed, you'd know it cable: you don't get to pay less by threatening to switch providers (I've seen people try to suggest that and get laughed out of the room), but by buying more from a single one.

Hence accident forgiveness is in the same marketing bucket as free credits for new customers: reducing the aversion to trying the cloud and putting more work on it.

And the switching costs, the biggest line item isn't building for multiple clouds (you can't be cloud agnostic: abstractions always leak somewhere, so you need to select a set of specific clouds to build for), but the cost of moving data between clouds. That's the real lock-in.

In the IBM, HP days it wasn’t about paying less per se by threatening to switch providers, it was getting more concessions (of which money might be one). Some big companies had both systems so they could play them off of each other. Time to order more hardware, who will kiss our butts more?

I’d be very surprised if that doesn’t still exist for the big boys. Though most of us are not big boys, and half of the biggest boys are cloud providers themselves.

AWS is something like an insurance service. We all pay more so that they can forgive the people that mess up xD

That’s not how it works.

We all pay more [sic] so data centers would be over provisioned, allowing quickly expansion when we need it. I’ve been in a few incident that were root caused to a cloud provider lacking capacity to provision more instances.

As that over capacity would’ve been bought and installed in the cloud provider’s data centers regardless of the errors that are being forgiven, not billing for those errors is a net gain to the provider, at minuscule cost, if at all, to other users.

Reply on news.ycombinator.com