Hacker News Reader

Comment by em1sar | Hacker News Reader

It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example).

For example, it's totally understandable that the alarms can be specified per region, why shouldn't it be like this?

Also the global AWS billing $300 alert seems to have worked but you were asleep as far as I understand. If it was a call-out style alert, then you would've noticed in the middle of the night and could've stopped it.

The only thing I agree is frustrating is this: > CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).

Maybe you can argue that you weren't made aware of this but idk... keep us updated

View on HN

What’s the case for not implementing an optional “shut down all my services at $spend and stay shut down until I intervene” ?

Honestly? Many people would enable it, forget about it, and footgun themselves on the other side.

Perhaps AWS should have "personal/developer" accounts that have this enabled by default and continually warn you about it, whereas "company/enterprise" don't have them.

I’d think if your rate of spending is >$50/hour then that’s nearly-always a bug. The only reason this conversation is taking place is because serverless “infinitely scales”. Autoscaling physical instances has a max limit for similar reasons.

I've experienced plenty of scenarios where costs have quite legitimately spiked.

Ultimately whatever solution you put in place, someone is going to complain about it. At least with the system they currently have in place they can reimburse customers. Whereas it is a lot harder to fix their reputation after they've automatically stopped production services.

Given how easily they reimburse customers, I suspect it's intentional - one can be "fixed after the fact" and the other can't - if your site goes down during a slashdotting and you lose sales, etc, there's no getting those back, but if you inadvertently run costs high, they can just refund/cancel those costs.

Azure HAS this hard limit feature already.

Ive seen nobody on HN, twitter, reddit complain about "my site was down during heavy business since i turned on hard billing setting". Not a single person.

However, I see frantic after frantic post of "I was testing something on AWS and it caused me a $X000 or $X0000 bill."

But as the posts in here are apt to suggest - you can always beg AWS support for a reversal. Great plan there.

The first is obviously customer error and unless you're posting to get laughed at, you're likely not to gain traction.

(Also one could make the "nobody uses Azure" joke here.)

Personally I think that much of AWS is "way overpowered" for the normal person/business, and you shouldn't be playing with it if a $X0k bill would be impactful (as likely other solutions are much better tuned to your needs and money).

If you drop a laptop, that's customer error. You break something or do something unintended that damages it, that's customer error.

When you are handed a tool that has multiple hidden guns and explosives inside of it, and ends up blowing your foot off is malfeasance of the people who handed the tool to you.

AWS is that tool. And given that Azure can implement these guard-rails and AWS chooses not to tells me all I need to know.

> Personally I think that much of AWS is "way overpowered" for the normal person/business, and you shouldn't be playing with it if a $X0k bill would be impactful (as likely other solutions are much better tuned to your needs and money).

Please compare and contrast this with "Learn AWS for furthering your career".

They 'might' reverse those fees, they might not. You are at their mercy and mercy is finicky.

That’s literally the point I made :)

> Many people would enable it, forget about it, and footgun themselves on the other side.

Yeah, but I figure as long as Amazon doesn't immediately remove stored data, the damage of the footgun would be minimal. Speaking for myself, of course, I'd rather have a short outage than an unexpected thousand dollar overnight expense. It seems so trivial that it's unclear why AWS would not implement this feature. The only explanation that makes sense is that they want these surprise bills to occur.

Because the downside for a company isn't "oh it was off overnight" it's "we finally hit it big and made zero sales because AWS shut us off".

Given how easily they reverse the bills, I suspect that they have a policy of doing it (perhaps a few times per account, something to prevent abuse) because they really don't want to trigger the above scenario.

Alternatively it's "we would have made a profit this month but a bug in this one service chewed through our budget in one hour". Sure, you might be able to get a refund, but that's no way to plan a business.

It's not really difficult. They just need a way to set hard spending limits. Probably on by default.

Unless you're a big company, "we stopped your function in the middle of the night" is a whole lot better than "we ran your function all night and you owe us $4k".

In my 25 years of running production services, I honestly cannot think of one company I've worked for that would have accepted their function being stopped in the middle of the night.

AWS already has a recourse for incidents like these: refund the spend. That is far more reliable than trusting an organisation can tolerate an outage.

The difference is none of the little sideprojects I work on are worth $4k in a month, let alone in a day. Obviously most companies want to spend the cash, but they should want as many programmers using AWS for sideprojects as possible.

It was more the "big" part of the "big companies" statement I was disagreeing with, rather than the "companies" part.

Sorry, I might have been not very clear, but AWS billing alert for $300 have been triggered only when it have reached $1,484 in charges.

If that alert triggered earlier (and $300 would have been triggered in an hour), my all accumulated charges would be only $400 not $4500.

So the hard learning here is that CloudFront charges takes time to appear on your bill, up to 24 hours.

24 hours delay in 2022 is egregious.

Reply on news.ycombinator.com