My Cloud Billing Screw-Up | Hacker News Reader

My Cloud Billing Screw-Up

masterpoint.beehiiv.com

11 pointsmooreds19 comments

At one company, we used Grafana Cloud for the full monitoring stack. They charge by unique Prometheus series for metrics. I wrote a rather small Go API to allow users to access some otherwise hidden data and added rate-limiting because the data itself was large. To figure out costs, I wanted to add monitoring to the handlers, so I added a middleware that caught all requests and logged things like the request path, response time, etc.

Sounds perfectly fine until you realize the internet is a vast space for people constantly scraping. I too left it over the weekend and came back to 70k unique series in our cloud account, pushing the bill well over $1k. What's worse is that Grafana is kind enough to not charge for these spikes, if you catch them before 48hrs. I caught it approx 50 hours later.

Like the OP, though, Grafana was nice enough to make it fall off after I explained the situation. Lesson learned!

Watch out for Grafana Cloud and that. All it takes is one dumbass to not understand cardinality of labels and histograms and you have problems.

Also on a Kubernetes cluster we hit 1.8 million series out of the box on prometheus so you can imagine how much much that would have cost. And we had one service double that due to some poor understanding by an engineer.

Anything that isn't constrained cost is a major financial risk.

Yeah, their prepackaged Kubernetes exports seem a bit excessive to me. We stood up a brand new EKS cluster, and it came in at 50k metrics, which was odd considering that's the exact limit before you get charged for additional metrics. It felt intentional.

I was in my first year of uni, I was coding an online multiplayer game in C++ and I wanted to test play with one of my friends mostly for shits and giggles to see how badly it would break.

I had deployed basic websites / servers with more managed platforms before, but I needed? more control to be able to host the C++ server.

So I found GCP, created a docker image, and got the server up and running somehow. We played for maybe 10 minutes before we ran out of stuff to do, and stopped playing. What I didn't realise at the time was that auto-scaling was a concept. I thought when there was no traffic then the server wouldn't work, and I forgot I ever deployed it.

Anyways, a month later I got a $400 bill, not nearly as much as some people have lost but for a broke college student it was a lot - especially considering I only used it for 10 minutes.

One time I was a happy customer of Raygun for error monitoring, and I enabled Raygun's new-at-the-time APM product on my API to monitor performance. I can't remember if the high sample-rate was the default or if I was just dumb, but I had a really high sample-rate configured. I was super happy to get so much visibility into my application's performance, especially query performance! However, I wasn't happy when I saw I racked up a $14k bill in just a few days.

Thankfully, they forgave the bill (thanks jdt!), but it still scared me. I was still at the point back then where a bill like that could've killed my company, or at the very least got me into a lot of trouble.

After that, I pretty much ruled out usage-based billing for my company as too risky. This was quite a few years ago, but to this day I still have no major dependencies that offer usage-based billing.

I see a couple of problems here:

1. "It was late and I was done for the evening so I didn't validate the change." - if I could use one sentence to explain what is my value as a DevOps engineer, it could be putting these safety pins in place. You shouldn't need to validate anything - it should be a part of the pipeline.

2. AWS is using extortion fees for things like NAT Gateway processing, egress traffic etc. Knowing that, and being aware that container images need to be pulled frequently, it does make sense to use ECR or any other internally hosted container registry. If you don't do that, you will spend that $1000 anyway, just over a longer period than a weekend.

3. Any changes on Friday evening - just don't.

I wasted two days a few months ago when Amazon randomly started charging us $1K a day for static s3 buckets with a few GB of data that hasn't been touched in like 4 months.

Turned out it was a billing error on their side that they would have probably completely ignored if we didn't notice it.

You also get billed for incomplete multi-part uploads! Those can build up quickly if you're using pre-shared keys.

This is not really a screw-up story, this is a "NAT Gateway is a racket" story.

One time I had the brilliant idea of writing recursive cloud functions, and boom! $4,000 in just a few hours due to a bug.

Luckily, my credit card had a reduced limit, and later Google Cloud forgave the debt as long as I promised not to do it again.

Strange. My Hetzner never Screw-Up.

I purchased a Hetzner machine but didn't get to it for a week. When I tried to configure it, I couldn't connect to it no matter what I did (reset, hard reset, KVM in, etc.). Turns out it was already (physically) dead. That was my introduction to Hetzner :)

That's why you need to pay way more for Cloud :)

I can relate in a way. I'm a happy Hetzner customer for over a decade. Now most of my clients are on Hetzner Cloud but before these were physical machines. So I just needed to remember all these mdadm commands to rebuild the array as a disk would fail every 4-5 years on each machine. But there were some close moments:

* Once two disks failed at once - fairly normal occurrence but they should know better and differentiate - fortunately the third one allowed me to rebuild the whole thing

* Once the motherboard failed (but they replaced it within half an hour or so)

* Once there were strange irq messages on the console so I just migrated the machine to another one

* I had a problem with a server once that it would not react to any standard reset attempts so I had to ask for a manual intervention - it worked but turned out the machine had a hidden fault so they replaced it

All in, I'm happy to have migrated to Hetzner Cloud as I don't have to worry about these anymore. I still have two physical machines but just for fun (as a Minecraft server for my kids and a Nextcloud setup for my ebook collection).

Adding here my own personal experience: I have a physical machine that runs Debian for Minecraft, a few static sites, a small Go API and a handful of internal tools and I've barely had to do any maintenance (in a year's time), no issues on Hetzner's side either.

I can't wait until we're all on IPv6 then NGW can go away!

Heh, it's kind of cute you think Amazon will just leave egress/ingress fees on the table for their very real "premium bandwidth". Gateways or not, they'll continue to squeeze out as much profit as possible.

Oh completely. That is a rounding error on our bill though. Their main income is not clearing things up I suspect.

Yeah...If age has taught me anything it's that I it's rarely a great idea to push changes at the end of the day. There's always that motivation to shortcut the validation a bit and every so often...this happens. If schedules permit, I find giving changes a final look first thing in the morning let's me catch things before they get committed.

Reply on news.ycombinator.com