Hacker News Reader

Comment by nvader | Hacker News Reader

Yeah, we probably want some kind of traffic light system:

Green means go Orange means finish what you're doing but don't start anything new Red means stop everything

And probably a special rule to permit stable, critical spend through regardless, the same way we allow police and ambulance to run lights.

View on HN

Stop everything is pretty damaging any real business though. Things were better in the era of VPSs. You paid for a fixed amount of compute, if you ran a stupidly expensive operation than it just maxed out your system for a certain amount of time and things slowed down. But you didn’t kill the service entirely and you didn’t have unlimited potential price

In this example, that would require the “big query” to have billing baked into its actual query runtime, which isn’t impossible, just not how one would design a query planner per se. Usually such services emit metrics of usage units, then the billing calculation happens in a completely different system taking into account discounts, promotions, contracts, regional and currency differences, etc.

Suddenly a database, a storage service or a computer service needs to be aware of the billing situations and make behavioral decisions based on the billing status. Again, not impossible, but something that suddenly promotes billing from an async/non-crucial background service that can be paused, replayed, adjusted by account teams etc, into a crucial hot-path service.

The way you'd usually handle that AFAIK is to have the service ask the billing system for a "reservation" in its native units, likely with an attached TTL. Then, the service would translate those units to U.S. Dollars (or possibly Indian Rupees), taking your plan, discounts, vouchers, contracts, grandfathered pricing and all that into account. It would then "lock" the calculated amount of money, denying the reservation if total_spent + total_locked > spending_limit. After finishing the operation, the service would ask for actual billing and free the unused units.

Yes, thats one way to implement it. It still moves that global billing system into a ring 0 importance with 99.9999% availability and performance requirement, where before it wasn’t even in the picture. Not impossible, just suddenly every single AWS (or GCP or Azure etc) has a global single point of failure that gates all their performance and runtime actions.

Not to mention having no way to network isolate such service as every single service in every single network boundary needs to be able to contact such service, and every service needs (more or less) same auth permissions on all user accounts it’s running.

Again, solvable problems with enough code, but not simple problems by any means if you operate at a large scale.

Reply on news.ycombinator.com