Firebase bill is usually $50, but I was surprised to see a $70k bill in one day | Hacker News Reader

Firebase bill is usually $50, but I was surprised to see a $70k bill in one day

twitter.com

117 pointsplainOldText158 comments

Maybe this has something to do with it?

https://x.com/tamarajtran/status/1867342095033258466

There's a really good reason that I don't link to my apps on this site (or most, for that matter). They are free apps, running on a shoestring budget.

But that whole account looks a little dodgy. The posts make it seem like one of those "I made $30K/mo, from my living room! You can too!" outfits.

People always underestimate how easy it is to sell $70k for $30k.

As long as you don't literally sell dollar bills, it can actually be quite hard.

And this is why so many in the industry mistake selling $70k for $30k as "product market fit".

I've had the very nice pleasure of always doing business with providers who would simply drop access to the server when bandwidth had been met or exceeded with my service.

But what's odd to me is that I don't often read about other people having the same experience. So... are all of you seriously with hosts that don't shut off access to your servers?

I'd rather that happen than suddenly get a bill I can't afford.

Run your cloud contracts (or anything possible to incur liability) through a separate LLC so if there are billing issues you can just move on.

The structure should look like:

"My App LLC" has contract with "My Host LLC" for hosting services. "My Host LLC" provides those services via a cloud provider. If "My Host LLC" racks up a 2 million dollar bill with the cloud provider and goes out of business then I just move to "New Host LLC" and carry on.

It's not as if the patients of a Doctor who fails to pay his office rent would become liable.

This is the entire purpose of LLCs.

I'm not a lawyer but this sure sounds illegal. Piercing the veil?

ah, yes. the "Shady Roofing Contractor" business model.

"don't worry, all the work has warranted by New Quality Roofing Services IV, LLC"

although, there's very little about that structure that's unique to an LLC. it can be done the same with a corporation.

i'm certain this only works when the amount of debt being ripped off (ahem in dispute) is not a lot more than the cost to sic a really good law firm on you.

LLC liability protections are not absolute. I wouldn’t be surprised if this would fall under gross negligence or callous indifference or some other legal umbrella that voids the liability protection. I also wouldn’t be surprised if there’s some fine print in the account agreement that creates a personal guarantee.

> callous indifference

i like this one. the ultimate law to subjugate everybody once and for all!

That’s not going to fly.

Firstly, because cloud contracts generally prohibit renting out the cloud services as-is - you can build a product on top of it but not act as a reseller of their platform.

Secondly, and perhaps most importantly because setting up separate LLCs with the goal of avoiding lawful debts is the demonstrates fraudulent intent and will get your LLCs veil-pierced.

What you mean I can't set up an LLC for every large expense I want to avoid paying?

You could, but anything you buy as that LLC would actually belong to the LLC and be subject to seizure and sale when the LLC doesn't pay its bills. That's my thousand-foot understanding of it, at least. If what you said worked people would do it all the time.

You're doing it wrong.

You need to be a billionaire, and then you need to threaten to sue anyone who dares to ask you to pay.

> cloud contracts generally prohibit renting out the cloud services as-is

Doesn't have to be as-is - a simple infra on top of it will do

> setting up separate LLCs with the goal of avoiding lawful debts

If normal bills are paid as usual and only this kind of crap is not paid, Amazon would have a hard time proving in court that the whole point of the daughter LLC was to avoid paying bills.

> This is the entire purpose of LLCs.

It’s not. A lot of weight should be put on the concept of “legitimate business”. Let’s say you decide to open up a car dealership and you use an LLC to buy cars then transfer them to another LLC and sell them. Then declare bankruptcy on the one containing the bills. You won’t be shielded, and you’ll likely go to jail for fraud.

The construction is to allow a business to go bankrupt with limited liability not to allow the expense account of a business to go bankrupt.

It doesn’t work that way.

When a dealer has a floor plan (to finance their inventory), the cars that are being bought by the dealer are collateral and have liens on them.

The lien has to be satisfied (bank paid with interest) to transfer the title to a buyer or to your theoretical “another LLC.”

Just like my car would have to be, for me to get a clear title to sign over to you.

Not everyone uses a floor plan, an unsecured line of credit would be the one situation where what you’re proposing would “work”

Floor plans are one of the reasons there’s only a few (used) dealerships in town I could walk into, plunk down cash, and walk out with a signed title that day. Because they’re transferring the title to themselves, and then to me, using the chain of ownership boxes on the back,and then sending that to dmv.

I didn’t know a lot about this other than the owner at my previous company had attempted to branch into car sales before he passed away. TLDR, we couldn’t even sell or dispose of the cars with good intentions and they all got repossessed.

Incorporating is good business advice in general and something you definitely should do but bankruptcy isn't always an option and those debts don't magically disappear. Unless you're planning on running an illegal or quasi-legal business from the start, avoid high risk strategies like this.

I'm not driving, officer, I'm traveling!

In the days before, there used to be a hard spending limit - I know this because of an old official tutorial video on youtube. Last time I checked, there was billing alert functionality and they were recommending creating a function that will halt your service when it hits the limit - the problem is, billing lags behind and you can rack up a hefty sum until anything is triggered.

Amazing service but its a scary one. Every time a social-media-worthy accident happens, they cancel the bill but I wonder how many are burned without recourse.

Maybe if your rack up something modest like 5K accidentally, its better to push it as high as you can and get it declined on your CC and increase your social media prospects :)

If you owe Google $70k, you have a problem. If you owe Google $70T, Google has a problem.

It’s weird how normalized over cap billing became acceptable simply because chargeable metrics were not collected/resolved until after the fact. Seems like an obvious gap in the process or a little bit shady.

The idea that an app will experience stratospheric growth so suddenly that costs must be allowed to grow 1,000x hour-over-hour has always been insane to me.

Why does Firebase insist on hanging the sword of Damocles over all of its customers? I’ve read so many stories before and experienced these fears the first time I was setting up Firebase…this has been going on for years

That's every cloud provider. At this point, I think they're actively conspiring to not implement billing caps.

I used to think this until I tried architecting out how you'd build a billing cap. I recommend it as a design exercise. It's easy to build a bad billing cap that would slow down services and cause outages, but it's basically impossible to build a good billing cap.

How is the service being able to answer the question "is there budget available for this action?" different from "is there authorization for this action?"

One example - Google Cloud network egress charges aren’t known until up to ~2 days after they happen. Since they can be obscenely expensive (eg $0.23/GB), they can make budget computation difficult.

What is the cause of this delay?

I don't know for sure, but based on my knowledge as a user, I'd guess it could be something like delivering usage logs from points of presence in the CDN. PoPs can go offline regularly, they're highly dependent on other people's networks, 2 days might be the arbitrary line that has been drawn that gets enough of them in most circumstances, while not being too annoying for customers.

Authorisation is much more cacheable than a value that inherently changes every single time you check it.

Also authorisation revocation is relatively uncommon, which means you can have a fast-path for approval, and then push only the revoked key IDs to just frontend servers.

> It's easy to build a bad billing cap that would slow down services and cause outages

When you've exhausted your billing cap, what else could it do? Either shutting off services or blowing past the cap seem like the only serious options.

For a lot of small businesses, I suspect that an outage is often better than risking a surprise 6 figure bill. But it depends on what your software does.

Also if the system shut down automatically when the budget got exhausted, there's a risk that a runaway backup process or something might accidentally eat through your regular budget and get the site shut down. For that, it might make sense to assign different resources into different budgetary buckets or something.

I'm surprised firebase doesn't implement something like that.

I am sitting on an algorithm for hard billing caps right now that seems it may have some holes, but gets close based on several very tricky distributed systems problems. Making a billing cap that doesn't amount to "just use one single gateway server" (serializing everything and introducing tons of latency) seems to be harder than building a database or a filesystem, and most programmers would never attempt even those.

> cause outages

That's fine. The major LLM providers work like this. If you're out of credit, or hit your monthly recharge limit, it stops working, bringing down prod with it if your product relies on it. Not heard anyone complain about the concept.

If it's really a problem for you, you can be all enterprisey and contact sales, then they'll be very excited to offer you extremely high limits and post billing.

This way everyone gets what they wants.

To be clear, the outages I'm referring to are not when you hit your billing cap. Try designing a billing system for a cloud provider that implements caps, while still retaining the performance necessary for the services you're providing to make sense, and without introducing huge, common, failure modes.

Seems pretty similar to distributed rate limiting. But it's much simpler to solve the common case of overspending on a single API: give each API the same daily limit with no communication between APIs.

Oddly, they have no problem shutting things off when the limits of their free plan are exceeded:

https://firebase.google.com/docs/projects/billing/firebase-p...

I am not sure how they can do that, but cannot let people set their own limits on their paid plans.

That cutoff is rarely truly a hard cutoff. The limits are often too low to have a natural test of that, though.

They could always make the amount over that is given due to their cutoff enforcement being less than perfect free, as it likely already is on the free plan. That would avoid the risk of unbounded bills associated with going on a paid plan.

Most of the cloud providers have a less-than-perfect cutoff. It's worse than the cutoff of the free plan, though, because the free plan can be slowed down to have better enforcement, while the commercial plans have performance SLAs to hit.

Limits Reached -> PubSub Notification -> Shutdown Sequence.

Because it's a free plan, the delay between 'limits reached' and actual shutdown only incurs the cost of providing the service during that brief period, not the potential liability of overcharging that might exist on a paid plan.

Is that really a problem though? Just don't bill beyond the cap then and leave the last few requests free, too.

Or write a disclaimer that the billing cap doesn't necessarily cut off at exactly that amount and that there might be an overcharge.

I am pretty sure most people would be okay with either of these options, we didn't need a perfect system, just one that works well enough

> it's basically impossible to build a good billing cap.

They don't have a problem implementing caps on a free tier. No one's asking for perfect, but they don't seem to care about even getting to the ballpark.

Either you want automatic scalability or you want caps. Scalability is hard with caps. Say your site selling stuff sees spike and scales up and hits cap, should the service degrade in way you did not plan for? Or go past the cap as you are still making money?

You solve this by opt in, not fancy engineering. There are two classes of customers - those that absolutely can't afford services be cut, and those that absolutely can't afford a 50k bill.

So you deploy an advanced technology known as a radio button to toggle which they want, throw a bunch of ToS & consent agreements about data loss / deletion at the ones opting for hardcaps....and done.

Also reminder that Azure has hard caps for certain account types. This is not a technical problem. They can do this, they just don't want to.

there really isn’t a conspiracy. a hard billing cap is at least one of: very difficult (even for faang) to implement without incurring unacceptable performance regressions, impractical as downtime has worse optics than high spend (given that high spend in this case is correlated with traffic, which is good), unnecessary by those customers who represent most revenue.

Setting max auto scaling is doable. Even if that doesn’t translate directly to hard billing, it would still help

I use fly.io. I pre-paid for credits and if they run out, things shut down.

No affiliation, just happy to use a provider with actual caps.

Wait till we see the crazy/unmetered AI bills.

These folk can't even get a stable billing process; the coming surprises will be awesome.

Look at how insane twilio is.

I set up automatic recharge of $20. A small amount because not much traffic. A bad actor got ahold of our api that didn’t have rate limit yet and started spamming Africa.

Twilio had zero issue charging my credit card every second. Literally I was getting a hundred emails and bank notifications a minute. Brex didn’t stop anything.

Twilio responded that it was my fault. Yeah. I sure 100% probably should have put in that cloudflare rate limit first. But…

How easy would it be for twilio to prevent this on any level? I need rate limits? How about you rate limit credit card charges. Putting $20 recharge limit should mean $20/day or $20/hr not literal unmetered right to charge as much as possible in 20 increments.

Twilio support sent me all this info about protecting myself from African spammers who use the technique to make money from SMS charges. You know what’s more responsible than informing me of this? How about blocking sending sms to country codes known for this from the get-go and optin to send to them.

it was clear the perverse incentives that encourage twilio to massively benefit from being insecure and easily exploitable by spammers.

Ended up costing almost $3k after bill adjustment when our usual spend was $5/mo. not bankruptcy level so after fighting with support just took it as is and learned my lesson. But twilio made *50 years* of revenue in about 10minutes from their own negligence.

It’s probably part of the business model. They rely on the African spammers to improve profits

I use digital ocean (a cloud provider) droplets. I know exactly what my bill will be at the end of the month.

Imagine if LinkedIn wanted to use Firebase to launch a new product.

I could easily imagine a 1,000x hour over hour growth as the social media grows.

If I was LinkedIn, I’d be very upset if Firebase pulled the cord when everyone was looking at the new launch.

Would you be upset if they pulled the cord at the limit you explicitly set beforehad? If not then that's not really the thing the people asking for a billing cap are talking about.

What does "pulled the cord" mean?

I pay for storage. Does this mean they delete my production database? They delete my s3 buckets? They delete my server logs? My message queue?

Then yes! I would be extremely upset. How do you expect the cloud provider to magically know what is safe to "pull the plug" and what is not safe?

More like make you servers publicly inaccessible rather than delete them. Or throttle bandwidth. Or provide a warning when the cap is approaching. Etc.

Exactly. Why the fixation on one strategy for handling this not so uncommon scenario. It is so common that handling it should be defacto.

This isn’t a pre-paid gas pump use, but that could be one way to present it. We all want to fill as fast as possible. And if your fill spout can handle top rates, you get top fill rates, until you close in on the hard limit. Then it trickles down to the metered drop. Then stops precisely where it needs to.

By accepting/requesting a hard cap, the provider can make clear that in order to be precise, soft caps will go into affect earlier and induce progressive throttling where applicable. If the throttle doesn’t catch the final milliliter or two of gasoline, before the pump shuts off, the provider can and should just let it go. It’s a loss, but comparatively a figurative drop in the bucket.

The other obvious route is predictive where prior usage guide the guardrails. Ordering two eggs is typical for a single meal. Ordering twelve is not. Ordering three or four is unusual for most but if you are a regular diner your habits will be observable.

Any of this predicated on the provider to want to do something. They seem to lack incentives at this point for making it easy. It is stories like op that I avoid well known problematic providers like Firebase who don’t respect and foster long term relationships.

Why do you pick unreasonable interpretations, when there are so many trivial, obvious, reasonable options? Even the literal interpretation of "pull the cord", meaning the power cord, would mean the servers shut down, but nothing is deleted. And why do you claim the provider would "magically" have to know these things? These can be communicated through user settings: "if bill exceeds X, do one of the following:

[ ] continue billing me [ ] throttle bandwidth [ ] gracefully shut down servers (data is deleted after 30 days of non-payment)

This is so painfully obvious even for someone that never deals with cloud vendors, that I just don't understand why you would pretend otherwise.

Almost all GCP services have quota by default that you need to manually increase.

Firebase is pretty simple to use, and possibly because of this, I've seen quite a few terrible implementations. I think it attracts people who only know app development, where security, scalability, authentication, etc, are not really a concern because you have exactly one user on one device.

It's easy to accidentally make a bunch of information world-readable, and if a malicious user gets hold of the right details they can simply read all your content out, or even start writing bad content in, racking up a bill in the process.

Whether this is what happened here or not I have no idea. And I don't think this is even really a problem with Firebase, just an indicator of the sort of developer who ends up using Firebase and their background.

It's simple to start but quite hard to master actually. It has plenty of counterintuitive concepts that even an experienced developer can get wrong and result in ruinous expenses or catastrophic security issues.

Especially on Firestore. For example, the access rules that you define on your properties are not filters(they are rules on what you can ask the system to do) even if they look like filters at first glance and the way it accessing data is billed is based on what has had to be processed to return the results and not on what the actual results were returned. This makes it very easy to create very expensive and compromised apps if you slip.

Also, unlike more traditional system, doesn't produce smoke so it passes all the smell tests and you learn about your mistakes once things go very bad.

That’s the third big GCP bill story I’ve seen in like 48 hours.

Over on Reddit someone got stuck with a 450k one.

Struggling to find the third thread again though. Maybe they deleted it

Also seeing an uptick in people suggesting insurance as the solution. Which seems insane to me but what do I know

The insurance thing also seems weird to me, what are you insuring against? It sounds like people are suggesting insurance against... cloud providers just deciding to randomly bill you, but that's not how it works.

Are you buying insurance against having a successful business? No one is going to sell that. Are you buying insurance against being hacked? That makes sense and does exist, but only solves one very specific version of this and isn't really about cloud billing, plus it's likely not accessible to the size of business (or individual use) who get bitten by this sort of thing. Are you buying insurance against incompetence using the services? You can get professional indemnity insurance, and I guess your own LLC could probably sue your own indemnity insurance in this sort of case, but you'd need to defend the decisions as reasonable at the time and not negligent.

They are just insuring against an unexpectedly large bill. That's how I read it. It doesn't even have to involve any proof or disproof of negligence. Just make it a purely financial transaction. You pay $1,000 as a premium such that if the cloud bill exceeds $10,000 the insurance will pay you half of that, up to a limit of $20,000. It makes sense as a financial product. Of course the numbers above are purely hypothetical.

Just because it's possible to state a hypothetical financial product does not mean there's a useful product there.

Why would I agree to pay up to $20k on a $50k cloud bill, when you could just choose to spend more on your cloud bill. I would be giving away free money and you'd get a discount.

You have to tie it to either incorrect billing, valid billing but caused by a malicious third party (i.e. hacking), or valid billing caused by negligence, and in the latter case you would need to prove negligence otherwise it's just a business deciding to use more resources and not pay for them. In the same way, fire insurance covers your house burning down, but not if you burn it down yourself.

Such insurance likely would require documenting full due diligence and being able to prove that you followed all instructions and known best practises at time of incident. Which likely would get the insurer out of most cases...

No it does not make any sense whatsoever.

If you wanted to sell such insurance, I'd be very happy to buy it for my upcoming $20,000 cloud bill!

I learned from this that there is something called “Errors & Omissions insurance” that professionals could buy. But in this case, it looks like the person who did the Tweet had hired a freelance software engineer through Upwork.

E&O insurance can be pricey. Lots of people skip it.

I have not been thrilled with Upwork, but that’s another story.

In general, I have found that you get what you pay for.

> Also seeing an uptick in people suggesting insurance as the solution. Which seems insane to me but what do I know

A lot easier to simply not use these cloud services.

Better yet, don't write any code at all! Never an over charge and your own accounting becomes super simple.

I'm sorta going for a middle ground. No internet facing usage priced big cloud...just back end stuff that is decoupled.

Eliminates risk of a denial of wallet like attack. That does leave risk of footguns & mistakes on my part, but that's a bit more manageable of a risk.

Does Firebase allow you to set up a billing limit, or is it one of those exciting cloud services where no matter what you do you're always one mistake away from losing your house?

Apparently she had a $20 limit, but, if my calculations are correct, $70k is more than that, which seems odd.

Somebody else pointed out that it's likely just an alert, not a hard limit, which checks out given Firebase documentation (https://firebase.google.com/docs/projects/billing/avoid-surp...), which has no mention of hard limits and explicitly warns you that an alert won't stop anything.

Ah, oops...

They have a documentation page titled “Avoid surprise bills”[1] but I imagine it’s easy for some developers to skip over that.

[1]: https://firebase.google.com/docs/projects/billing/avoid-surp...

https://firebase.google.com/docs/projects/billing/avoid-surp...

> We don't turn off services and usage because although you might have a bug in your app causing an increase in spend, you might just be experiencing unexpected positive growth of your app. You don't want your app to shut down unexpectedly when you need it to work the most.

Frankly, I don't see anything on that page that would actually prevent a surprise bill.

> A budget alert sends an email whenever your project's spending level hits a threshold that you've set. Budget alerts do NOT turn off services or usage for your app.

It's not an automatic hard-stop so you could still screw yourself over pretty badly with runaway spending.

eg. OpenAI enforces strict limits until you spend a significant amount of money.

My guess is that OpenAI’s margins are much lower so they aren’t in a position to forgive or have people skip out on big bills.

For Firebase, their costs are probably pretty marginal.

From my experience, cloud LLMs are still being used by most systems as an "additional feature" with fallback to alternative basic functionality, or some other form of graceful degradation. On the other hand, there typically isn't a good fallback to having your main DB go down.

Once you do spend a significant amount of money, you'll find that the limits you can set yourself might not even work.

We have a multi-organization OpenAI account and I had set a $4k/month limit on one of the child orgs that was being used for a R&D project. Got billed ~$20k for the project one month and complained that it clearly allowed us to exceed our soft and hard limits. We were told that the limits (which you can set and act like they are a real thing) don't do anything if you have a child organization set up. They refunded us after some persuading and I know it's just normal growing pains for an organization that is undergoing rapid growth and maturity, but was still a little surprising that even the "hard limit" didn't do anything for us!

(Note that this was last year, so this bug is probably long fixed as they have redesigned their portal multiple times now)

OpenAI functionally have one API, it's easy to limit spend on one API. It's much less easy to limit spend across hundreds of APIs, and resources that have ongoing cost (like VM hours).

I understand joking about this and the possible downsides, but there are good reasons why cloud services are set up this way: 1) it's impossible to bill at scale and exactly cut off service usage globally when a target is hit, and 2) most companies don't want a single point of failure like a misconfigured budget to bring down their production services.

The answer is probably quota management, where a limit on the number of VMs or size of database or something, caps the worst-case scenario, and where it's arguably easier to monitor an approach to that quota as it's more granular than billing.

Personally I think cloud providers could have an explicit "hobby mode" that limits certain things in such a way that the spend can't run away like this, with the trade-off that they're not really production grade in a sense, but then again those accounts are probably worth anything so I understand not building that out. That said, whenever I've seen one of these things happen, it always ends with "FooCloud said that as a one-time gesture of good will they would write off this accidental usage", so while briefly scary, maybe this is the system working fine overall?

Azure does actually have the ability to force-kill your resources when you hit a certain billing threshold, but only for things like free trials and student accounts. The instant you switch to a regular pay-as-you-go account that functionality disappears for uh... reasons.

https://learn.microsoft.com/en-us/azure/cost-management-bill...

That's good, it makes sense it would be for only those sorts of accounts. Imagine building a billing system that could always do that, any time you accept a request, you need to check against a database if the user has hit their billing limit or if you can charge them for it. It obviously can't work. I imagine the way MS make it work is probably to slow down resource consumption for those accounts dramatically, and then just take the hit on overspend, knowing that it will be almost nothing.

I understand that a precise billing limit is probably impossible at that scale, but if they have the ability to send you an email some time after you go over a limit then they have the ability to automatically pull the plug at the same time rather than waiting for you to notice the email and do it yourself. They just don't want to.

I'm sure people would accept a best-effort system where setting a billing limit for $100 means you may be billed $140 because your spending overshot the limit before the system noticed. It still beats the alternative of waking up to a $20,000 bill out of nowhere.

So I've stored some data in a bucket, the storage time goes over my budget, what do you do now? Just delete the data? I think most people would think that's a terrible outcome.

Let's say the answer is yes, you just delete all the data as I can no longer afford it... delete operations are billable, so do you charge the user for those deletes or not?

Let's say the answer is yes, and you bill the deletes. What happens if too many deletes are required and suddenly you're at 2x the bill cap? Now you can't document the bill cap as being able to go over by up to 1.5x. This may be unlikely, but customers use cloud services in weird and wonderful ways.

This is just one resource type, there are many different resource types on a typical cloud provider, each with multiple axes of billing, each of which has hard decisions to not just be made, but documented and communicated to customers in such a way that they understand the impact. Oh and also it's the "I just put my credit card in and go" crowd who you have to explain it to, who aren't engaging in sales conversations, not those on business contracts who might actually listen or read the documentation.

It's not at all obvious to me that this is preferable to just having someone look at these incidents on a case by case basis and seeing who should be refunded.

This is an insane take, there are many options here. The idea that the reason to structure billing this way is anything OTHER than that it's the most profitable way to do things is ludicrous. It doesn't matter if you can construct some other plausible reason, as long as it's the most profitable way to operate why would I believe the cause to be something other than profit maximization.

What alternative do you suggest? I'm not saying providers should delete data when hitting a cap, but rather that this is one example of why you can't cap spend.

It is possible to combine all the billing axes for things like this so that, but when you do you get Dropbox, or Google Drive. The explicit value proposition of cloud hosting is paying for what you use, and generally granular services are lower margin and more commoditised than higher level services.

You could just throw an error, freeze services and do whatever "let someone handle this on a case by case basis". It's absurd to suggest that the customer doesn't want to have the option to prevent their spend from growing by multiple orders of magnitude without a human in the loop. Sure maybe deletes are a billable action (also absurd and fake), but having the options to say "hey I can't spend more than X, cut my service if my spend + cleanup would cost more than X" is absolutely doable and something many people would want.

If you think that deleting all data (blob storage, block storage, VM states, caches, etc) is preferable to a surprise bill, then I don't think there's anything we can debate here.

I agree that there isn't anything we can debate here, but it's because you're making straw men. There's a middle ground here between getting charged 4 orders of magnitude more than you expected and having all of your data deleted that you're obtusely refusing to consider.

If writing data to a bucket would push the monthly cost for data storage over the budget, the write should fail, not succeed and then delete something else to get the data back under the limit. Why would you even consider doing it that way unless you're specifically going out of your way to write the worst possible form of billing cap?

That's clearly not user friendly. Users cannot predict the amount of overshoot. So we are back to square one. The user could wake up to a $1,000 bill or $10,000 bill and all data deleted, and the cloud provider can just say "oh we run our billing limit enforcer job on an hourly cron schedule but your account requested many machines very quickly so you still incurred $10,000 worth of charges." A precise billing limit is impossible, and a fuzzy billing limit with a precise error bound is the same thing and also impossible. Now we are back to a fuzzy billing limit with an unknown error bound.

> That said, whenever I've seen one of these things happen, it always ends with "FooCloud said that as a one-time gesture of good will they would write off this accidental usage", so while briefly scary, maybe this is the system working fine overall?

I suspect that if they tried to sue you over an unpaid bill, they'd lose over issues of proper notice to you. You can't actually bill someone for a service they didn't want and didn't ask for; that's why the wash-your-car-in-traffic people are considered scammers.

I'm not a lawyer, maybe it wouldn't hold up in court, but I imagine that good will is the driving force, even if way down the line they may not manage to actually reclaim the money. My experience of working with cloud providers is that they are almost always happy to take a short term loss (extended trial, more test resources, refunds for accidental usage, etc) to get an account that is going to grow and stick with the provider. This does make sense, customer acquisition and churn are expensive, and recurring revenue is great.

> it's impossible to bill at scale and exactly cut off service usage globally when a target is hit

This seems unlikely to me. What is the technical reason for this?

To do this you would need to check in with a central billing service every time you want to charge, and that central billing service must keep a consistent view per customer to ensure it can't spend over the cap.

This is not too hard if the billable event is, say, creating a VM. Creating the VM takes a few seconds, it's easy to add a quick API call to check billing. But what about the next hour, when you charge for the VM again? You now have the number of VMs checking in every hour (worst case, at the same time), and you need to handle all those hourly checkins consistently per customer.

That's still probably easy enough, but what if it's not a VM hour, but an API gateway request? Now on every single request you have to check in with the billing service. API gateways need to be very low latency, but you've just added a request to that process that possibly needs to go across the world to check in with the billing service running on another continent.

What if the billable resource is "database query seconds", and now you need to know how many seconds a query is going to take before you start it? Oh, and add the check in time to every database query. What if the billable resource is streaming video, do you check in on every packet you send out? What if it's CDN downloads, do you have every one of thousands of points of presence around the world all check in, even though the point of the product is to be faster than having a single far away delivery node?

There are bad workarounds for each of these, but they all mean either the cloud provider losing money (which, assuming a certain scale of customer, is too expensive), the customer over-spending (which assuming a certain scale, could still be waaay over their budget), or slowing down most services to the point that they functionally don't work anymore.

All these arguments seem very much like throwing the baby out with the bathwater - I don't think we should pretend it makes sense to say "if we can't have perfect billing cutoff down to each individual api call we shouldn't have a billing cutoff at all". You've listed super achievable ways to prevent a $50/mo spend from ballooning to $70,000.

Additionally, it feels hollow to not have billing cutoff at the same rate as authorization would cutoff if they shut off my account.

I understand, and I do think an approximate cut off would be good for some users, but I don't think it solves this problem for a few reasons. What constitutes a bill shock is wildly different between users. Is a $50 bill a shock? It is to me with an average AWS bill of $0. I don't think you can set absolute or percentage values that make sense, and you can't let it be configurable because this gets into issues like the SLAs on billing logs arriving, the overspend becomes the margin of error in the cloud provider's systems.

The other main issue is documenting this. Google Adwords I believe has an overspend concept, i.e. if you limit your billing to $100, they might still go over it. The problem is that it's limited to 2x your bill, which still bites people. I only know about this from reading HN and Reddit posts complaining about it!

What I see in this thread is tons of people saying "the ideal of perfect billing cutoffs with no over-runs is impossible, which is why there are no billing cutoffs" even though I've also seen lots of people point out that - to simplify - something is better than nothing, here.

A $1k overrun past your billing cap is still way better than a $50k overrun - the cloud vendor is more likely to get paid in the end, and the customer is more likely to come away from the experience looking at it as an 'oops' instead of a catastrophic, potentially-finances-ruining 'i'm never touching this service again' incident.

There are plenty of really challenging problems in computer science and we solve them with compromises every day while hitting demanding targets. If a SSL certificate expires we expect it to stop working, and if it's revoked we expect the revocation to take effect eventually. But it becomes a situation where these guarantees benefit small companies and independent developers but we suddenly can't solve similar problems?

Fundamentally speaking if you can't afford to check against the billing cap every request, check every 10 requests. If 10 is too often, every 100. If 100 is too often, every 1000. Or check once per minute, or once per hour. Or compute a random number and check if it exceeds a threshold. The check can even be asynchronous to avoid adding intermittent latency to requests.

Any of these are better than nothing and would stop the bleeding of a runaway service incident. It's unrealistic to expect small companies and independent developers to have someone on-call 24/7 and it's also unrealistic to expect that if you sell them $100k worth of stuff they can't pay for that they'll actually pay you somehow.

You don't have the individual vms check in. You have the VM coordinator report how many vms are running and get back an affirmation that it can cache until the next reporting period that the total is not over budget. If over budget, coordinator begins halting services.

API gateways are similarly sending metrics somewhere. The coordinator can be the place to ingest that data and send the aggregated info to billing. If it gets back over budget, start halting endpoints. etc.

Or do it within the billing service, but fire off a shutdown notification to the coordinator of whatever service created a billing record if over budget. Same idea.

Basically, batch, amortize and cache work. Same as every computer problem. You establish some SLO for how much time your services can continue running after an overage has occurred, and if that's a couple minutes or whatever that will cut out like 99.99% of the impact in these stories.

Solving this for any one resource type, or one billing axis, is absolutely achievable in the ways you've suggested.

Solving this across all resource types and billing axes however is a different problem. You can't cache the notion than a VM is under the billing cap for an hour if there's another service that push spend over the cap within that hour.

You're right that you could establish SLOs across everything and minimise the amount of monetary loss, in theory, but at scale (as some resource types necessarily bill infrequently, as customers are spending more per hour), I suspect even this breaks down.

Then there's still the issue of billing at rest. Do you shut off VMs? That might be an easy question to answer. Do you delete their storage though? Harder. Do you delete blob storage? Probably not, but you've got to swallow the cost of that until the customer decides to pay again.

I strongly disagree that this is in any way defensible, despite that both Google and AWS do it. You should be able to set a limit, and even if they can't cut off exactly at the limit, they should be able to cut off when you hit (say) 2* your limit, or in the absence of a limit, perhaps issue progressively louder warnings until you hit 10* your usual spend, with the ability for you to give affirmative consent to uncap if you're hoping your app will suddenly take off. Nobody needs to lose their house or their life savings to some teracorp just because "it's hard."

I totally agree. You should be able to choose to have things shut down rather go past a limit. I don’t have a single client that would choose $70k versus, say, a couple hours downtime while we figure out what is happening with a resource that is going crazy.

This is where I question the risk of serverless. Although, now that I think about it, while my one client’s EC2 instances are essentially capped in terms of capacity and spend, we also use S3 etc. I suppose it would be entirely possible to accidentally write a huge amount of data to S3. But again I would rather get warnings from the app that writing to S3 has failed due to limits than get a huge bill!

But you also pay for data at rest in S3. Should S3 stop storing that data while you figure things out? Should they bill the customer for the deletion of that data as they normally would?

I don't disagree on wanting this feature, but it's just not something that's possible to implement in totality when you dig into the details.

Could it work based on deviance from norms? Like a setting where you say, I’m okay with up to two standard deviations from our typical write volume but that’s it?

Or, alternatively, just let me set the size of the volume. Treat it like a hard drive?

In terms of your point about the data at rest, part of the issue is that we get a bill once per month, and that’s probably when we would notice it. Of course there is probably a Cloudwatch alarm or something we could set (I assume) but there’s so many damn services…

This is sort of what quotas enforce, and most cloud services have quotas. You ask for an increased quota if you reach the current one, or the system might sometimes automatically increase it if it thinks you will need it.

This all trades off though against the possibility of bringing down one of your customers when they are hitting peak sales on their website, which is a very bad look.

Hobby mode is using something like Hetzner instead of messing about with "cloud native" nonsense. For the $50/mo they expected, they could get an absolute monster like a Ryzen 5 3600. Cloud native stuff is for when you need to be SOC2 compliant or something and want to minimize access to everything. Clouds charge you worse than enterprise pricing (enterprises negotiate discounts).

> it's impossible to bill at scale and exactly cut off service usage globally when a target is hit,

How much does the problem change if you remove the word 'exactly' from here, though?

Like, I don't mind if I end up paying a couple of extra bucks. Or even tens of bucks! Some people might not mind hundreds or thousands, or even more depending on their scale.

But blowing out several orders of magnitude past my usual monthly spend is the problem I'd like to avoid.

AWS has a "hobby mode": they call it Lightsail.

We are going to continuously see cases like these until the vendors implement functionality like:

  Hard Spending Limit
  If you configure hard spending limits, then upon the sum of money being reached, the service will be stopped until manual user action is taken. This can result in service outages.
  Spending limit: ______ $
  I have read and agree with the terms: [X]
  [Confirm]

Which is likely to happen never. That's why I mostly use traditional VPSes.

One of the reasons Supabase is getting so popular now.

Absolutely. Even getting a super sweet deal, like “$1/month” can backfire, because suddenly you are on the hook for $299 /ysatly, because the three month subscription defaults on a high premium once it’s over.

Not what would be the consumer friendly choice, which would be to charge “same as before”, i. e $0.

>Which is likely to happen never.

Correct. Because it would change the dynamic with enterprise customers.

Right now CEO/CFO is forced to accept unlimited open ended billing. If there was a way to set a hard limit companies would utilize it as part of their annual budgeting, not just hobbyists. That would be bad for big cloud's profits.

That’s exactly the kind of nightmare scenario that inspired us to create Prelude (https://prelude.so/). Authentication flows are targeted by fraud like SMS pumping that leaves companies with super high bills that make no business sense.

Our API helps apps take control of their SMS verification costs by proactively blocking fraudulent and suspicious verification attempts before they ever trigger an SMS. On top of that, we let users set daily limits as an additional safety net, so surprises like this don’t happen.

Clearly the problem here is using SMS as a verification method instead of an authenticator or literally just an email.

> Upwork engineer

Arguably worse than an average 2025 LLM.

Someone came up to me at CES peddling outsourcing teams. I told him we just use Copilot.

If he had a good sense of humour he'd have replied "so do we"

Yeah there's your problem. Hiring a random person and giving them unlimited access to compute resources is a bold move to say the least. If you're gonna work on something like this, do it yourself or hire someone who you can really trust. That won't totally prevent it but it's better than rando Upwork contractor who will move on to the next gig

Damn, Just learned that firebase budget is not a hard limit. This is bad. Is there a way to setup a hard budget limit? if not, i think I will need to replace firebase for my web app.

As far as I can tell, no cloud products outside of consumer-facing SaaS allow you to set hard limits or automatic stops. The best you can do is configure spending alerts (and hope you're awake to see them).

I've found it pretty nerve-wracking, then, to attempt to get some hobby projects online. I don't like writing blank checks for my hobbies..

That's because (spoiler) SaaS is made for b2b not for regular people

Actually my complaint was about using IaaS as a hobbyist, but sure.

This happened to me previously where enabling a firestore backup service almost doubled my bill. In my case the google support people were very gracious to wave that off, it took some time but it happened.

Had a good experience with them.

Though I wish taking backups on firestore was easier.

Ugh

I still have a couple of apps running on Firebase. I can't wait to get rid of those.

In 2016 Firebase seemed the holy grail for frontend devs and I deeply regret ever using it.

hopefully dev has insurance?

but anyway, this is partially why I spin up an LLC for every app i make... just declare bankruptcy and kill it

You're not wrong. An LLC and similar aren't perfect, but it gives you an enormous amount of legal bargaining power compared to not having one.

In what sense is it not perfect? As long as you don't commit actual fraud, creditors can't "pierce the corporate veil" of an LLC, can they?

It's not technically fraudulent to use your personal bank account, per se. You need to keep great records though. But from what I understand that could be enough to consider the veil pierced. Also need to make sure you don't personally ensure any debts.

What the sibling said. You also have to be sufficiently business-like in your projects, according to legal advice I got. If you slap an LLC on your obviously hobby project, they might be able to make the case that it wasn’t a legitimate business operation. That is fraud isn’t the only way to pierce the veil, I was told.

This is the way.

That's a really cool idea. Would you mind sharing more?

In particular, do you use something like Stripe Atlas for that? And if so, is there any impact to your account with them when you declare a bankruptcy?

Yep, while it's a few extra hundred bucks, Stripe Atlas is just super easy and you get tons of credits and stuff through the partner perks. Plus I've always enjoyed using Stripe for payments.

Apparently you can get it 50% cheaper via Microsoft Startup Hub:

https://www.microsoft.com/en-us/startups/blog/trusted-partne...

How do you create your LLCs?

I've only ever used Stripe Atlas, but there's other alternatives. Atlas gives you a bunch of free credits with their partners and is pretty quick to get through, but you might be paying a hundred or two premium over doing it all ad hoc.

Would you know if I already started a project on my personal account can I still move it to a new LLC account?

Also, assume I already have an LLC then how do I indicate this account is an LLC? Can you sign up as an organization on Firebase?

I’m not positive… I would imagine you’d need to transfer ownership. Like if I have an iOS app under my personal dev account then start an LLC and someone sues me for whatever reason, I think it would be hard to argue the LLC actually owns the app, but IANAL. You can generally sign up for most of these services as a team or business but I’m not familiar with firebase. It being Google though I’d assume they support business accounts

A pain point will be you’ll need to wait for your new LLC to get a DUNS number + apply for an Apple developer account. (Whilst you’re at it, set up Apple Business Manager too.) Realistically you will want a domain for that LLC that has email, so set that up too.

You sell it to an LLC and draft a bill of sale/invoice.

I’d only feel comfortable with a new account at Firebase explicitly in the LLC’s name, and keep my personal name/identity far away from it.

It’s not hard to get an EIN (a few clicks online and free), open a bank account (my credit union turns this around in a day or two), fund it, use debit card as billing method.

I do a few clicks on the Secretary of State’s website, pay $99, and the certificate shows up in my email the next day.

Doing a search/replace on the operating agreement for the new entity name, filing it away somewhere, etc. typically takes more time.

Probably a silly question, but what are the apps she built? Just curious what they are. Thanks.

I notice that the post is now gone. Looks like she locked her feed.

Probably because of this discussion.

Go get a $50 unlimited traffic VPS from IONOS.

This is ridiculous (for Firebase). There have been countless episodes like this one happening, as well as the amount of people asking for a hard-block when the spending reaches a certain limit (currently there's only a notification that is triggered, which is anyway lagged behind the actual execution). Making a mistake can happen to anyone (especially for a platform that targets itself to new devs) and makes me seriously consider leaving the platform.

To Firebase: either you must automatically condone such mistakes or implement the requested feature.

To anybody else: Does Supabase or other platforms offer this?

tldr: stored 1pb in a GCS bucket. this persons twitter account is about how they have hacked growth and have had 3 number one apps, crazy growth, etc. well, I hope they can pay.

also, billing caps really don’t make any sense. a competitor could easily exploit that to take your service down. best to you know, architect your stuff correctly… speaking of architecture, creating a paas where you have a hard billing cap would pretty much obliterate performance as you would need round trips each time you do anything in order to confirm you’re not over the amount.

Did they jump from zero to 1PB in one day? Or is this something like GCS usage only shows up on a Firebase dashboard view once a month?

(I'd look this up myself but replies on Xitter seem to be Muskwalled)

yes, apparently on october 18th alone they managed to rack up a single petabyte, which honestly sounds implausible, but who knows what they were doing.

The thread said the app was only running for one day, so I think it wrote 1PB during that one day.

So, better if competitor will make you bankrupt?

It would be better if your competitor made your app or project a popular hit.

I'd argue that depends entirely on your business. For example if your app has a physical component (e.g. a gadget you sell), having an unexpected inflow (by orders of magnitude) of costumers may very well exceed your capability to fulfill your side of the contract by months or years. And having to cancel or delay things infinitely can also be very damaging.

So it entirely depends on what it is you are doing.

I always use a virtual credit card number (like Citi provides) everywhere which has a daily hard limit.

Good advice indeed, but paying is only one part of the problem; you'd still owe the bill.

Provides precisely zero protection against this.

Even if your card declines or you used a prepaid card...you're still legally liable for the full amount.

Reply on news.ycombinator.com