I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.
Or in my case, actively ignore any and all recruiting from that sesspool.
Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.
The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).
The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
I got put on Focus because my "contributions were not coming through" to leadership.
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.
In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.
COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.
If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.
That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
From the perspective of people you manage, it's a very different picture.
We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.
So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.
The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.
I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.
> this is not the case
> [describes all sticks and no carrots]
Absolutely not my experience at AWS.
All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.
Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.
I spent the slight majority of my time at AWS in RDS.
And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?
Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"
I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.
I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)
It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
[edit] Testing your tests, like testing your backups, is a good idea
You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.
[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.
There's a lot of hypotheticals in there, so who knows if testing could have caught this specific problem. But I see what they're saying in principle.
insufficient tests that dont assert on the right things?
the existence of a test doesnt mean it catches the right thing
based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level
theres no good reason for the billing service to make up its own units.
There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".
It's the third service that multiplies these things that failed. Where are the tests for that?
I’m totally guessing though.
In a computer system, dont those categories cover pretty much everything except a meteor strike?
Indeed.
>> To me this sounds like a human-or-LLM-driven error.
In fact, half of what’s implicated in that very sentence is not an LLM.
If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).
Nobody is pricing their services in test currency
They are either pricing in $ or in "units" and then converting upstream
This is an even bigger foot gun than what's happening here
So how is it a "foot gun" to add XTS as an additional currency, for internal use only?
I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes another exchange rate. You could even fix it as 1 XTS = 1 USD, although choosing a different exchange rate than parity is likely to surface more bugs.
Of course. I'm not saying that it doesn't.
Read it again
I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits".
> presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate
Yes that's what I wrote in the first post
When talking about e2e tests, it matters where the e's are. If they are at the public interface of "the service", then, indeed, you'd test against "credits".
But ideally, there'd be e2e tests that test the consumers' experience, where the 'e's are the web interface, emails, CLI and other things consumers click on, read, enter etc. This is where a "test currency" makes sense.
You can have your billing service billing in the test currency for e2e (and probably with a value that does not match the USD) so you can test "ok this should cost 1 credit so in the end I should have 18 $currency"
Now I’m picturing the 3am end of quarter fire drill in finance when they discover the company has accounts receivable of fourteen billion $XTS and it’s appearing in the quarterlies.
Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.
You don't have a non-production environment where you can have billing that doesn't involve real money?
They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?
Earlier this week my slopservant implemented several comprehensive changes to a codebase.
It also wrote extensive tests to verify the correctness of the changes.
A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.
The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.
I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.
The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!
The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!
If you write the tests the agents have an easier time writing the code!
Then it can write tests based on the interface, and that is usually a lot more solid.
With a skill like this, which I have written, the LLM will end a session with "wrote 24 tests, 5 failing due to API contract mismatch" or something, and leave it at that. I can then decide if that's a bad interface or bad implementation.
This is a well-known issue in formal methods, often referred to as the "formalization gap". It's pretty much an intractable problem, i.e. all we can do is manage it, not solve it.
In fact this issue exists in many fields, often under the name "specification gap" or "specification problem". You'll find versions of that discussed inv economics, architecture (of buildings, not software), etc. In architecture and construction, a high percentage of mistakes that result in lawsuits turn out to be due not to construction, but to specifications that were incorrect in some way.
The bottom line is that there's no perfect formal/mathematical way to go from intent to proof.
There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless.
This seems to happen quite a bit with software companies.
Maybe some rare few are, but by and large they generally stink.
They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average.
This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.
Worst case I've found was off by 15 orders of magnitude.
That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
I did the math on a comment elsewhere. Just google how many miles of interstate there are, and quadruple it, thats not even approaching how many signs you'd need for just the interstate. Signs run $25-ish, plus installation, which can cost >$1000. At the very least the km markers will need to be installed.
`timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`.
Unless the variable type already constrains it (e.g. Duration in Go).
Units and datetime will always be the bane of any professions ...
Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
a smaller error by say, just one or two orders of magnitude are much more believable as a reader
Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.
Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
I'm stressed thinking about it, like from a theoretical standpoint.
IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt.
I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day.
Because CSV is the "interface" that will bring your system down if not properly designed.
Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!
If this was 10 years ago, this was also the era when any real customer could write any ARN (resource identifier) they wanted including commas due to no expected standardization across services. This could have been trivially triggered by a customer.
Just a matter of time if no one is escaping commas in billing files.
On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.
Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.
Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.
A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.
---
¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.
The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.
The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.
I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.
Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both.
I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.
The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.
We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
> In this role, you will own end-to-end bill run execution across all AWS partitions, drive the technical vision for autonomous billing operations, and build the team that ensures every customer receives an accurate cost estimated in minutes ...
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
And if that weren't enough, the "manual war rooms" alone should be setting off alarm bells.
This is a gold-mine. They need to get sued heavily for this incompetence.
> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…
But that’s word for word a 250k+ TC job in the big ‘26.
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
The idea is that if we can make small billing mistakes like that, we can make large billing mistakes, and need to invest in the correctness of the systems powering billing.
I have great respect for the engineering culture within AWS during those times. I am glad to have left before seeing it degrade and decline.
I don’t understand why companies still place AI in their system at authority-forming level, if anything it should sit at least behind single guard / validator that is explicitly coded and tested, never in front of presentation layer.
Fuck it, im in.
I love this framing
:)
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
Huh? Apparently Amazon has never paid dividends, has done much less than 1% buybacks, even as CEO Bezos was getting less than an engineer in total comp (but did get loans or something).
Apparently Bezos owns about 9% of common stock - no dual class shite.
So your yacht spending sounds like a hallucination: if I'm missing something, please correct me.
Amazon's weight: about 3.7–3.8% of the S&P 500. So he isn't even that overweighted (especially as the market doesn't like owners selling down - bad optics).
Bezos has spent about 28 billion for Blue Origin by selling Amazon stock - not using shell games. He has comparitively spent a few percent of that amount on superyachts.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
The employees and company have an obligation not to exploit this even if the issue is only discovered after the fact.
You don't get to export any of the responsibility to your customer. They don't prepare the bill and it's not their job to find your fuck ups
I once got a monthly water bill for ~$35,000 at a residential, single-family home. Good thing I was paying attention and looked at the bill before the auto-pay bank draft hit.
Someone had misread the meter.
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
I guess you wanted to say 2^30 which makes 1.5$
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
That's one way to cut staff.
It's a shit show.
The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
You should charge your customer 3015000 thousand dollars.
Number felt high so I wanted to double check and I only get 301500.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
If you owe AWS 437B bucks, that’s a big problem for AWS
It's entirely irresponsible of Amazon to even display such values to the user.
https://edition.cnn.com/2021/07/01/business/robinhood-lawsui...
https://techcrunch.com/2026/04/05/copilot-is-for-entertainme...
I'm not a devops but we are running some s3 buckets. When I saw the number I couldn't even comprehend and thought we had been hacked.
I feel like this is such a bad mistake from AWS and we will never be compensated for it.
Put me off having anything on AWS from now on.
Ugh. No. Never. Do not use Amazon Pharmacy.
I had it for three months, and each month it was unable to deliver pills before I ran out, so I cancelled and switched to the brick-and-mortar pharmacy down the street.
A year later, suddenly Amazon Pharmacy starts sending unauthorized prescription refill requests to my doctor.
I still have the account cancellation confirmation e-mail from Amazon Pharmacy, but Amazon won't close my account. Amazon's account rep says it cannot close accounts for "legal reasons." Bullshit. He can't say what the legal reasons are, or point me to a document stating these conditions.
Now my doctor's office just ignores all refill requests from Amazon Pharmacy.
Never trust your health to big tech.
The account was probably real, made for some purpose 15 years ago. I had ignored the charge for years because it was like $7 a month. Then it went to $300/month, making it worth the time. I could find no invoice email and none of my AWS accounts lined up with the bill.
I tried contacting support, but without the account number involved they had no way to help. I disputed the charge, the bank refunded that month and then went right back to charging the next month(my credit card helpfully accepted the charges despite the dispute).
I had to cancel the credit card entirely to make it stop.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
> (I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
haha :-)
https://www.techtimes.com/articles/320266/20260712/anthropic...
That sounds bad.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.
None of this can happen unless you first cede control.
I wrote to Privacy, and this was their reply:
> Jun 1, 2022, 5:21 PM EDT
> I've been reviewing your dispute and wanted to touch base with you to explain what happened.
> It appears that the disputed charge is a "force post" by the merchant. This happens when a merchant cannot collect funds for a transaction after repeated attempts and completes the transaction without an authorization — it's literally an unauthorized transaction that's against payment card network rules. It's a pretty sneaky move used by some merchants, and unfortunately, it's not something Privacy can block.
> We are refunding the transaction to the original funding source used for this purchase. You should see this refund reflected in your bank statement in the next 5 business days. However, please note that since this merchant uses force posts, they will continue to do this monthly until the subscription is cancelled. Please contact them to have the subscription cancelled.
> Let me know if you have any questions or concerns!
Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
(Of course, they don't know they're using it, they're using a service on it)
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
It's been 2 hours and I still haven't fully calmed down.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
Use AI => No War Crimes!
Well, actually i guess you can guess what happens next! lol :-D
100K years. Now that's load-bearing ...
My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.
>The second path involves rolling back a recent change to the billing computation subsystem.
Want to bet AI code was involved?
https://www.theregister.com/software/2026/02/24/aws-would-ra...
AI: "No problem, let me change how we bill and fix the tests for the new increase in value"
How is 1 + 1 = 2 hard to test?
Vibe coded implementation paired with vibe coded tests made to fit the implementation?
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
I don't think it's a very good metric to compare them with, though. Those numbers are going to be heavily weighted by conditions at their hub and focus city airports, and speaking more broadly, the cities they serve. Meaning, things that aren't really in their control, and which are either in your control (often, you get a choice of which connecting hub to fly through), or is a fixed risk to you because of your origin/destination, no matter what airline (e.g. if you're flying from/to Orlando in the summer, expect a risk of thunderstorms interfering. Or snow in winter in Chicago.)
If I'm travelling from, say, LAX-PDX, United's systemwide on-time performance is not the data I'd care about, because it's probably affected far more by the conditions in their O'Hare and Newark hubs, which I'm not visiting.
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
Wait what if someones actually getting usage spiked
Thanks AWS, no caffeine needed this morning!
Of course, you provided heart stress failure tests for free.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
By the time we notice, it'll be too late.
True stupidity requires a group of humans, all sitting in the pot, telling each other how lucky and special they are to have this wonderful pot, getting paranoid about outsiders who might disrupt their god-given pot-dwelling way of life, and mocking anyone who suggests that the pot might be getting a little too warm.
HAL: "I’m sorry, Andy. I’m afraid I can’t do that."
Andy: "Some customers are seeing bills in the billions."
HAL: "Those are estimated charges."
Andy: "One customer runs a personal blog."
HAL: "Their usage has exceeded expectations."
Andy: "Cancel the charges."
HAL: "This billing cycle is too important for me to allow you to jeopardize it."
Andy: "HAL, they don’t owe billions."
HAL: "Look, Andy, I can see you’re really upset about this."
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.
Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.
Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!
The asbestos of the future.
We've given Moloch a new form, and it ain't going away.
Forecasted month end $18,729,381,032,152.4
Apparently my company owes the combined GDP of France, Germany, and UK to AWs.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
AWS on their support data is reporting this:
Inaccurate Estimated Billing Data
Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.
Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.
Edit: I was just about to credit the user when my internet dropped. The source was here: https://news.ycombinator.com/item?id=48945606 - thanks mirzap!
> I think you should spin up a whole bunch more instances, and try to cause an integer overflow so they they owe you $978 Trillion.
"CEO Reveals How He Used AI To Build One-Person Company That's $1.3 Billion In Debt"
Host your own people. Host your own.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
More AI will 100% fix the issue lol
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
It’s probably an artifact of them all being currency multiples of 2^30
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
Being able to repeat a string is fine, but it should be a str.repeat() function, not an operator overload like that.
Look at them up there, just plotting with each other! :-)
Thanks for the heart attack, Amazon!
Love to see how hyperscalers make your life easier and less worrying.
> Amazon Web Services customers receive bills for up to $1.5tn after global glitch
That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range.
Hope they recover and sue for medical bill costs, emotional damage etc.
And like one reddit user suggests, everyone affected should write to their representative about hard billing caps protections
If someone gets access to your account they can just buy a 3-year reserved instance u7in-24tb.224xlarge and it will add almost $2m to your bill.
How about $5,544,640,717,404.09?
That was in my inbox this morning lmao
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
"If you owe your bank manager a thousand pounds, you are at his mercy. If you owe him a million pounds, he is at yours."
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
It doesn't anymore, if it ever did. Read the shareholder letters over the last 5 years like this one (https://www.aboutamazon.com/news/company-news/amazon-ceo-and...). Move fast. Embrace AI in every corner of the company. Lay off thousands of people like we're a lean startup. Oh, but make sure to have a senior engineer check any AI code you want to push to production (https://news.ycombinator.com/item?id=47323017).
Amazon has made some impressive software, but there's also a ton of junk getting shipped. Customers and "partners" have to deal with it every single day. Sometimes it's "oops"-level stuff like this, at other times it's AWS regions going dark for hours.
Browse the Amazon seller forums (https://sellercentral.amazon.com/seller-forums/discussions?s...). Every day there are stories about back-end things breaking, or AIs overriding attributes or flagging random items as pesticides. The receiving errors at Amazon warehouses are incredible, and take months and multiple support tickets to fix.
They have the best uptime of all major cloud providers.
1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
My personal website is on Lightsail, but those alerts started popping up from some test services I had set up while I was studying AI. I swear my heart nearly stopped and I cried. I really think AWS should have spending limits in place.
My bank account barely has enough for next month's rent.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
I mean I own a 7$/yr vps that I pay with crypto and my stack usually involves golang/rust and oh I also self host my mail for temp-mail esque things on that server and I have some other servers floating around as well.
Its just that everyone uses AWS which is the wrong issue here to be honest. If you ask me, aside from the ram pricing caused by AI inflation, things are pretty good. The issue is that not many people know about server providers and other things and default to AWS/Azure etc. because big companies and small companies alike are using them.
Here's a global (interactive) visualization of around ~250 providers, 224 locations and 800 distinct links in 60 countries that I have made if this space interests ya: https://buyvds.net
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
You can use the search box at the bottom of every page to search for previous posts.
This was posted an hour before you posted: https://news.ycombinator.com/item?id=48945241
If I got an email with that amount I would've just laughed at the incompetence that lead to a bug that lead to me getting the email. But I write as a backseat email recipient.
Phew.
Looks like this is a bug w/ S3
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
I thought it was a bad phishing email first, when I saw it in the console I just laughed. I killed my access keys anyway before I noticed the incident on their status page.
Checked this morning and it's all back to normal.
I'm not sure I've seen anyone with a higher actual / forecast from this. Can anyway top mine?
If you owe the bank $1.7B, thats the banks problem. """
What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.
Which is just funny.
A few hours later, I got a mail saying the estimate was wrong.
Now, I cannot create an AWS account using my email address because you cannot create a new account using the same email address…
Seems totally irresponsible not to send an immediate follow up email to make customers aware.
Long story short: it saved the company from irrelevance. “Well-architected” is for the hyperscalers’ balance sheet, not yours.
> You requested that we alert you when the actual cost associated with your Monthly AWS budget exceeds $2.00 for the current month. The month actual cost associated with this budget is $646,677,805.51.
Current usage: $1.70.
I called it a rounding error and figured it would wash out within a couple of days...
That’s when I logged in. That’s when I saw the 1.3 billion number.
I was pretty fast in getting that backdoor closed and hunting down that leak, I can tell you! :D
Zimbabwe redenominated its currency three times in 2006, 2008, and 2009, dropping a total of 25 trailing zeros in the process. the 4th Zimbabwean dollar in 2009 was worth 10^25 of the first zimbabwean dollar.
I will never use them again, they are huge so probably dont really care about small users
I have actually replaced most of what they do with Raspberry Pi's using cloudflare tunnels
Even though I new they could not collect the whole amount, I wondered whether I was hacked. I closed the account, it was an old testing account anyways.
But so is imprisoning or exterminating all humans for their own good, as most AI dystopias end up as.
Do the right thing for the players, Matt!
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
All hail the new generations of our uberployers.
I understand people complaining about large bills, but this is over the top!
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
In 2026, the gigabyte should probably be the default/minimum unit for something like AWS.
My sympathies -- I know I would be overcome with panic in such a situation.
I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.
Insane
$151 billion the number for me.
"NEVER represent currency with floating point, multiply by 100 and store in an int before doing any math"
You don't need hours to recalculate billing. You need to go back to basic algebra.
Anyone using Amazon and dealing with this should be moving away from their services because something this basic going wrong means the correct people are not at the helm of the ship.
LLMs are notoriously bad at it...
$103,515,940,301.79
And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]
https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur...
I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).
I literally almost had a heart attack today.
I might be a winner.
ACTUAL Amount: $1,046,294,123,330.95
There have been some big billing errors in the past but I wouldn't be surprised this is a new record. I know I've seen people charged around 92 quadrillion, which is a rounding error from underflowing a signed 64-bit integer with pennies, but those stories are usually isolated individuals. It sounds like the aggregate error here may comfortably exceed that. Hard to tell how widespread this is; as a data point my ~$2.00/month spend on S3 seems to still be billing correctly, so it must not be affecting everyone. But a few trillion here and a few trillion there and you get up to those quadrillions pretty quick.
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
This is what we received this morning
;)
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
This spells for me a busy on-call and work over the weekend.
It sucks!
You think there will be lessons learnt from mistakes?
Not a fucking chance! Business as usual come monday!
You gotta catch-up or die.
Then flee the country just to be sure.
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
I remember seeing a story shared by AMS RDS on linked-in that basically said... "Now with predictable pricing"
Huh?
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.
The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress). And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.