Single day Firebase bill for $100k | Hacker News Reader

Single day Firebase bill for $100k

serverlesshorrors.com

23 pointstodsacerdoti14 comments

Interesting site. It would be good to differentiate between Denial of Service (DoS) and Denial of Wallet (DoW).

The contributors are running into one of the many common tradeoffs of Serverless, an inability to set spending caps.

I wouldn’t go along with the last line conclusion of the original post - “done with cloud”.

It might be old fashioned, but I really like Infrastructure as a Service (IaaS) rather than Platform as a Service (PaaS). Spin up cloud compute hosts as required, and avoid anything platform specific as much as possible. It’s easy to end up married to AWS, GCP or Azure because the cost of a migration project is always a bit too high to pay off.

My use cases might be smaller scale, but I currently like using Terraform and Ansible with EC2. My Terraform setup is quite specific to the AWS environment (EC2 hosts, VPC, subnets, etc) and would require modification if I jumped ship, but much of my detail is in the Ansible playbooks, which simply expect Debian Stable host(s) to configure with my chosen stack. I currently front all that with Cloudflare, and use the cache and built-in bot protections along with rate limits and WAF rules to try to mitigate against abuse that could run up a bill (such as excessive egress).

I'm in this boat also.

Being able to migrate at low cost (and in little time) should be a high priority for any software infrastructure with availability expectations.

Especially with providers with very reasonable bandwidth costs (~€1.19/TB at Hetzner) beyond the first 20TB.

I also use Terraform, and it does lock you to one provider. Switching provider means switching the complete set of resources. They probably have the fundamental concepts in place (a DNS record, a VPS), but things diverge somewhat quickly, e.g. when it comes to virtual networking, SSH key management, external storage.

Keeping a redeployment script to a secondary cloud in your back hand means you don't have to sit and translate Terraform once the shit hits the fan.

I'm currently migrating my Terraform to Terranix, which just replaces the HCL with Nix expressions; since they both compile to JSON, using Nix lets you generalise across providers at the cost of a more complex language. But you also get things like arbitrary conditionals, e.g. "if X then provision Y else provision Z".

Ansible sounds neat for bootstrapping the servers, but it runs a little short on the ongoing maintenance because it isn't declarative, so you get runaway state divergences.

Also, sorry for hijacking this thread to talk about our lord and savior. :-D

I'm the guy that got hit. Feel free to ask me questions. Planning to do a series here or on reddit about what actually happened when the dust settles a little bit more:

Latest convo is here: https://www.reddit.com/r/selfhosted/comments/1kdxqwj/burned_...

Are you sure your project is done for? It seemed kind of neat in hindsight. Why do you think so?

This is due to the way cloud is monetized being different than the costs of the underlying resources. The asymmetry of the cost for attackers requesting a file being $0 and the cloud being per request and byte transfered makes it easy to rack up large bills for little cost from attackers. With proper competition it would also be cheap to serve such requests, but cloud providers are a rip off.

It does seem crazy that the default state for Firebase, CloudFlare et.al is not to alert the account owner when out-of-the-ordinady usage charges hit, say $5k above "the norm". We're not talking a service interruption, just a simple heads-up email to confirm that all is well.

They don't need enterprise quotas for new firebase projects. Set them low, let us turn them up when quotas are hit.

For instance I was seeing 35GB/s (GigaBYTES, not Gigabits) per second of cloud egress. I believe 25GB/s is the default for each region where you have a bucket.

Who knows what other insane quota landmines there are out there.

Would be quite difficult to migrate to self-host, there is no open source version of Firebase APIs, and while there is alternatives like Supabase, it is a significant rewrite of both frontend and backend.

Main issue is the realtime database (old school legacy stuff). Month of work to migrate to something else and replace, but maybe I'll do it.

How does the average person recover from this? One screw up and it’s bankruptcy?

Not sure what their finances are looking like, but they seem to have had a premium service with paying users.

The service author said this in regard to their bill:

> GCP seems to finally be budging with regard to the bill. They acknowledged the DDoS and are running it through the bureaucracy. I do have some confidence that they'll make this right, but I took destructive actions to stop the charges (deleting buckets). I did have a mostly complete backup of customer data on another cloud, but this has destroyed small business side hustle, where I built a community of over 100,000 users over seven years.

https://www.reddit.com/r/googlecloud/comments/1jzoi8v/ddos_a...

They left a message on their site: http://simmer.io/.

And just yesterday, they launched their new side hustle: https://stopuncappedbilling.com/.

This is basically accurate. Google is working with me on the bill. I don't really want to give details on that, as the conversation is ongoing.

Hint: it's not easy and far from resolved.

Company was "profitable" if you count the 3000 hours of engineering I did as free, haha. Paid firebase bill with a little bit extra. Beer money, not money money.

crazy

Reply on news.ycombinator.com