Ask HN: Did your AWS bill literally double from last night? | Hacker News Reader

Ask HN: Did your AWS bill literally double from last night?

268 pointsent10168 comments

I just logged into my AWS account and I noticed every charge has doubled for my September usage, literally overnight!

I called AWS and they said this has been reported by multiple people and they still don't know how many accounts are affected, did this happen to you too?

View on HN

I opened a service ticket. My ec2 instances have somehow been running for over 950 hours in one month. I can go back and see my instances run for 744 hours (what i'd expect in a month) in previous months. Amazon invented more hours than actually exist in a month for billing purposes. Bezos is a genius! I should have thought of that!

Came across this long back: 2 + 2 = 5 for very large values of 2.

Maybe Bezos read this recently and decided to use large values of 24 to compute number of hours in a month.

EDIT: deleted

EC2 instances are now billed at per-second granularity, with a minimum charge of 60 seconds.

I just received this email from Amazon on Thursday afternoon, this may be related:

"We are notifying you that we incorrectly issued an early invoice for your September AWS usage on September 18th. As your card was charged successfully for this invoice, we are currently processing a refund for the unexpected charge. The refund generally takes 3-5 business days to be received by your bank and some banks may require additional processing time. Your full monthly usage will be invoiced through our normal billing process on or around the 3rd of October. We will send another notification when your correct September invoice is available. We apologize for any inconvenience caused by this error.

There is no action required from you, but if you have any questions, please contact the AWS Customer Service Team by visiting: https://aws.amazon.com/support, and opening a case by selecting 'Create Case'. You can then select ‘Account and Billing Support’ , followed by ’Billing’ for the Service, and ‘Charge Inquiry’ for the Category."

Yeah, I got the same email.

Mine is showing as roughly twice what I'd expect it to be on the billing home page. It looks about right on the cost explorer though.

I got an alert for crossing a billing threshold only five hours ago, as expected for this time of the month; now the total is just under double that.

It's quite an intriguing error. I'm struggling to imagine how it might have come about.

if i had to guess I would say that some usage data was piped through the billing system twice. Given how may people are experiencing this I hope this will be fixed in a timely manner.

Yes, seeing the same on the Billing & Cost Spend Summary; but the correct amounts are displayed in the AWS Cost Management dashboard [0].

[0] https://console.aws.amazon.com/cost-reports/home#/dashboard

Yes. This is the case for us too. Cost reports is showing accurate information.

I was also seeing a surprising cost of about twice what I expected, but since I rarely use AWS I was assuming I had just calculated it wrong somehow. It's maddeningly difficult to see your usage history of services -- I still can't figure out how to see what my EC2 usage was this month. There is something called the "cost explorer" which I can't use yet, because apparently it takes 24 hours to set up (?), and it's not even clear that it will include my historical activity when it's ready...

AWS usage reporting is utter crap. The only usable data for me has been the actual billing table. But that is only monthly.

You can make it dump monthly billing CSVs to s3 daily. You'll get a new CSV each day that contains the month to date so it gets bigger every day until it resets the next month.

The good news is that is that Cost Explorer is quite useful once it's on.

There is something called the "cost explorer" which I can't use yet, because apparently it takes 24 hours to set up (?), and it's not even clear that it will include my historical activity when it's ready...

Cost Explorer itself also charges per use. I'm not kidding:

https://aws.amazon.com/aws-cost-management/pricing/

The API charges per use, but not the cost explorer dashboard itself.

All those dashboards use the APIs underneath. The cost explorer dashboard certainly does incur API calls on my account.

no it doesn't

They probably have cost explorer data in a different system and need to index your history before it can be used.

It will include your historical activity.

Yes, just saw this too. This is showing across all service types for me. Anything that is billed based on a unit of time has exactly 2x the consumption than has happened so far month-to-date.

My website bill rose to exactly double. It's still only a dollar, but that's twice what I was paying last month!

"isitfit" (https://isitfit.autofitcloud.com) is a downloadable command-line tool that calculates billed usage of EC2 instances from the number of running hours over the last 90 days. You could reconcile it against the EC2 section in your AWS bill. To download it, use `pip3 install isitfit` and then just run the command `isitfit`. It's free and open-source at https://github.com/autofitcloud/isitfit

Edit: On a side-note, this week is the global climate strike week (https://globalclimatestrike.net/). Consider running `isitfit --optimize` to identify underused EC2 instances, downsize them, and use the saved money to save the planet.

Yeah something is crazy.. I killed a bunch of service content and my cost STILL seems to be going up.. My bill for this month (even after the reduction) is going to be almost 85% higher than last month with the same level of activity.

I wouldn't go killing important servers if I were you. I'm showing a bill for 950+ hours for a bunch of services that I only have one instance of each. Since that's not possible (there aren't that many hours in a month), and Cost Explorer has the correct numbers, I'm assuming that there's a mistake that's causing double-counting on the billing dashboard.

Is anyone still seeing problems like this? I found duplicate entries in our Parquet CUR reports, but not in the CSV versions (we export both, for reasons). The duplicates were from 24 September beween 00:00 and 13:00 UTC and only for Usage and DiscountedUsage. Not limited to EC2, though, it was Redshift, CloudWatch, and a lot of other services too.

I wonder if AWS is prepping for a pricing hike but someone was trigger happy with the big red button :D

This is absolutly not related, but on a funny note, I got subribed to prime forcefully last month.

I'm highly suspicious toward Amazon behaviour now. This kind or news only flurish the idea that Amazon has taken it's first steps toward being a total a company.

Whenever I hear news like this, part of me feels bad for all the programmers working very long hours to fix the problem. You can be sure that a lot of pagers started to go off as soon as the reports of billing issues started to come in.

'pagers' Now that's a word that I have not heard in a long time...

You can tell that I haven’t worked in an on call position in a while...

PagerDuty makes a lot of money https://www.marketwatch.com/investing/stock/pd

My partner still gets a pager for on call at a hospital, she gets a phone as well.

A colleague is a former AWS employee - my understanding is nobody wants to touch the billing code for this reason. I think that also explains why the billing interface leaves a bit to be desired.

I got a response back and my time was corrected (quick glance).. I will be more thorough from this point on. But for me the quick and dirty is comparing to the previous months bill.. Normally is enough, except for this month.

I'm showing double the amount of hosted zones in Route 53. Kind of odd

Mine doubled. Got a non sensical reply from AWS. Now back to normal.

Same here. Just checked my Org consolidated bill.

Every service looks to be about double, However if you go to the Cost & Usage report, it looks right.

I usually get billed 54 to 58 cents per month. It's showing me 70 cents for the current month, for a projected 91 cents!

We were just about to submit a ticket when the forecast/breakdown returned to normal. Looks like fixes are rolling out.

About an hour ago I received a refund receipt from AWS for the extra charge. So it seems like they are addressing the issue.

It is a mistake in the billing console.

Are you at an account anniversary? What does the service broken down billing say your biggest cost is?

No issues on bills of two orgs I have access to. Cost Explorer shows data as it should too.

I think it might have charged people for free-tier eligible usage as well

I've seen it happen

We had a problem with malware mining Bitcoin on our AWS instance a few weeks ago:

https://www.cybereason.com/blog/new-pervasive-worm-exploitin...

Oh no somebody ran the cron job twice.

Call me, aws. I can help. I know cron jobs.

Don’t call it that. It’s a Lambda.

From 1 to 2€? Maybe...

Yep, exactly double.

Yup, I am seeing double as well.

Thanks for the heads up. Our AWS costs were definitely up almost 30% for this month! Checking billing details...

Yep this happened to me.

i can't wait for this COE

There was recently an error where AWS send the bill twice and then sent an apology e-mail. It may have something to do with this.

Lol will have to check for a coe

Is there a word for cloud billing anxiety yet?

I had an AWS account I wasn't using, but I didn't want to get surprised by any bills, so I set an alert to let me know when the bill was more than $0.

Sometime later the alert went off, so I anxiously jumped into AWS to see what kind of disaster was happening, and found that I got billed a few cents... for the alert itself.

I closed it altogether after that. It's better for my sanity to just use a new email address and sign up again than to worry about surprise bills.

EDIT: It's entirely possible that I set it up wrong, or set up the wrong thing, but in any case an alert was all I was trying to do, and as far as I could tell that's what happened. I must admit I rage-quit rather than investigate too deeply.

> Sometime later the alert went off, so I anxiously jumped into AWS to see what kind of disaster was happening, and found that I got billed a few cents... for the alert itself.

They charge for billing alerts? I've never used AWS in production and I suppose it's a miniscule cost, but, wow, that seems really pedantic to charge for that.

AWS is a gigantic exercise in figuring out how to gouge customers. Case in point: charging API gateways per HTTP request.

I don't necessarily disagree with you, but I don't think that's a strong example. Charging for utilization is the most obvious way to price a service, and http requests are the most obvious way to track utilization of an API gateway (or any load balancer).

Which other service provider puts a price tag on each HTTP request routed through a reverse proxy?

Some service providers such as Hetzner don't even charge for bandwidth up to 20TB, but somehow AWS found a way to hike the bill per HTTP request.

I've never used AWS so perhaps I'm mistaken, but api gateways normally provide extra functionality on top of just proxying the request. You can still set up a server listening on port 80 or 443 with a public IP and make requests to it without being charged per request.

You can, and if you want to use Lambdas, you can also route to them from ALBs, which has a bit confusing of a pricing model for comparison, but has no per-request cost.

Oh, ALBs. Now that's a mess. You'll either be charged per new connection, per concurrent connection, per byte, or per rule evaluation, depending on which is highest any given hour.

https://aws.amazon.com/elasticloadbalancing/pricing/

> Load Balancer Capacity Units

I'm baffled at the complexity of just ELB pricing. Do AWS customers just not care?

Hetzner does not charge for bandwidth anymore starting from half a year-ish(?) ago. Unmetered gigabit.

Charging per request for something that has a cost to provide is reasonable (and it's a tiny charge in this case, $3.50/million or 0.00035 cents per request).

Otherwise there's no incentive for application developers to optimize their usage.

In my last company we charged per unit of work, but it was possible to send a lot of requests to do a tiny amount of work -- some customers hammered us with millions of requests to do each unit of work. It wasn't malicious and it worked with their workflow, but we lost money on those customers due to all of the infrastructure required for the request routing.

The company eventually started charging a nominal fee per request, which resolved the issue.

> charging API gateways per HTTP request.

You sort of had me in the first part, but you lost me here.

Charging $3 per million requests for a HTTP interface service?

How else should AWS charge for API Gateway?

(I suppose it could be bandwidth, but that doesn't make sense as API Gateway deals only in small messages; unlike ELB, S3, etc.)

Reply on news.ycombinator.com