Accident Forgiveness
fly.io
fly.io
The number you see on your bill is increasingly calculated by running some black box algorithm on top of the billing events your resources generate. Was it accidental or not? What is a "weird" deployment vs a normal deployment? By what factor should the spikes on your billing graph be smoothened? None of this can be deterministically calculated from the pricing page. And there's no way for you to do these checks yourself before deployment because you have no idea what this logic even is. So you are entirely at the mercy of the provider for "forgiveness".
Who wants to bet that some provider is going to launch "AI cloud billing" within the next year?
Keep laughing, I can switch clouds in less time than you take for figuring out cloud costs.
For me the sweet spot is to have all compute in Kubernetes and stick to open source or "standard" (e.g S3) services for your auxiliary needs, but outsource it to the cloud provider. Fairly easy to move somewhere else, while still keeping the operational burden down.
But agree that having e.g all your databases in a proprietary solution from the cloud vendor seems sketchy.
AWS has very nice one-to-one mapping of K8s serviceAccount with IAM roles.
We used some cryptography-centric GitOps patterns to eliminate any human hands beyond a dev environment which also helps IAM be easier (but of no less reasonable granularity and quality).
> the jump to some much cheaper bare metal hosting is not that far.
Heh, at a consulting firm I was at not too long ago, all the K8s nodes were sole-tenant on the cloud providers these K8s nodes were on. Intra-cluster Cilium-based Pod-to-Pod networking across cloud-based datacenter sites has been super smooth, but I have to admit I'm probably tainted/biased by that team's uncommon access to talent.
I think for that particular firm, "K8s tax" measured in fiat currency is negligible and anything on the human side their people upon hearing "K8s tax" would respond with some ridicule along the lines of "someone doesn't know how to computer".
To be fair, most of the commenters on HackerNews should use something like Heroku.
And also factor in (1) the claim that most cloud growth is ahead of us, eg. moving large customers from on-prem to cloud, and (2) it would be terrible policy to try to charge existing customers more than new customers.
Companies do this very, very often. This is part of the reason why they have a "call the sales department for special pricing" option. They can give large contracts a nice discount to get them onboard, then slowly (or in some cases, if they really think they have you hooked, not so slowly) ratchet up the price. This is common in both B2c and B2B businesses.
If you store a lot of data, transfering it is a big deal too, of course.
But really, computers are computers. It's easy to move software between them. It just takes motivation, effort, and time.
In business terms, you just said "cost, cost and cost".
Alternatively, when Stitches managed to ambush your level appropriate character alone in Duskwood
To roflstomp is to defeat someone so easily and completely that it becomes so comical you'll be ROFL.
Part 1 is bandwidth prices vary tremendously by location, but clouds would like to have customers in expensive regions too, and if US overcharges at 100x, and Brazil overcharges at 25x, the customer will only have to pay 2x for bandwidth in Brazil. Not a lot of cheap hosting in Brazil, from what I've seen, but there are a lot of users in Brazil, attracting cloud customers justifies putting more cloud hardware there which benefits the cloud.
The other part is that bandwidth is easy to measure and broadly correlates with general usage. On a shared system, you can't really meter watt hours, but you can meter bandwidth. Bandwidth charges are how the difficult accounting is reconciled so that the cloud can charge enough to hit their desired margins. If bandwidth was just a cost plus, other things would cost more; if everything had to be cost plus, accounting would be much more difficult for everyone (and it's already pretty non transparent)
If you actually look at your contract, no cloud provider is going to contractually obligate themselves to forgive your bill, and you shouldn't be planning or predicting your bill based on it.
I've rented a VPS from a vendor for going on 20 years now (Holy fuck I'm old) and I've never once been surprised at the bill.
That said, yes, just using a VPS vendor is the easy way to stick to the good parts.
I can't imagine a zero-page book selling well...
They also offer VPSes.
https://community.fly.io/t/region-specific-machines-pricing/...
If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for someone from their PR department to see it. Only then will they finally waive the costs.
Good on Fly.io for trying to handle such situations more sensibly.
A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
They really aren't. There's a huge world out there beyond the HN front page.
If you have to be cloud, do dev in one cloud and test/prod in another.
I know, I know, easier said than done.
Hence accident forgiveness is in the same marketing bucket as free credits for new customers: reducing the aversion to trying the cloud and putting more work on it.
And the switching costs, the biggest line item isn't building for multiple clouds (you can't be cloud agnostic: abstractions always leak somewhere, so you need to select a set of specific clouds to build for), but the cost of moving data between clouds. That's the real lock-in.
I’d be very surprised if that doesn’t still exist for the big boys. Though most of us are not big boys, and half of the biggest boys are cloud providers themselves.
We all pay more [sic] so data centers would be over provisioned, allowing quickly expansion when we need it. I’ve been in a few incident that were root caused to a cloud provider lacking capacity to provision more instances.
As that over capacity would’ve been bought and installed in the cloud provider’s data centers regardless of the errors that are being forgiven, not billing for those errors is a net gain to the provider, at minuscule cost, if at all, to other users.
Generally, I tried to make exceptions to grant the refund. It sucks to get these bills and they can be quite scary.
Unless you have a real chance of bursty traffic I suggest going with compute that has more predictable costs.
But many folks don’t need the elasticity at all. So you should factor that into your architectural decisions.
AWS are very well-known for bill forgiveness. It’s not something set in stone, but if your bill explodes accidentally, even due to a mistake you made, they will normally forgive it if you ask them. You don’t need to go running to social media at all.
1. After a DDoS attack, someone got a $100k bill from Netlify for his static site and after he asked to have it waived, they generously reduced it to $5k. Only after his posts about it blew up did Netlify waive it completely. [1]
2. Someone got a $1k bill from AWS because lots of people made _unauthorized_ requests against his empty S3 bucket. AWS did agree to waive it immediately, prior to any social media posts. [2]
I probably just remembered (2) as "that ridiculous billing situation involving AWS" but got the details of what exactly happened mixed up with (1).
So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.
If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with.
Surprise billing is never good for customers, only the business.
We agree completely about surprise billing.
If it were critical infrastructure, or monetized in a way that brought in revenue to cover the charge, then maybe I don't want it to shut down despite skyrocketing costs, but that's hardly the only situation you could be in.
Secondly, the best solution is to simply stop everything. But now the customer has to cold-start their entire infrastructure, which may actually cost more than paying the bill.
Thirdly, it is likely that customers will set a billing limit and then forget about it years later. Suddenly, they've got a complicated infrastructure setup spanning the globe. They finally hit a scale where they hit their billing limit that they had completely forgotten that Bill configured in their early days (who doesn't even work there anymore). Suddenly, the entire global infrastructure is shut down in the middle of the night.
That's the gist of what my friends said.
https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-y...
Although even then, "we will only threaten to charge you $100k without meaning it" isn't much of a reassurance.
There is also usually some "soft" limit Y < X that you don't want to exceed, and don't plan to exceed, but you'd rather pay >Y than face an outage.
But a hard limit would have to be set to X to avoid that outage, and if it gets exceeded, you'll face a bill of X and an outage.
So what a customer would actually need is to specify both X and Y, with the rule: If the cost would exceed X, then terminate it early so the cost doesn't actually exceed Y.
Sounds complicated to implement, but then, the current practice of waiving the bill is complicated too if you tried to formalize it.
(For the sake of this discussion, I'm ignoring all the technical difficulties of terminating a high-availability service at all.)
I definitely get the idea of: I don’t want to spend X so if it looks like I will, terminate service at Y. But I think that’s a special case of the general situation, I want to know how much I’m on track to spend, right?
But I don’t know much about this at all. My whole experience was accidentally getting my own personal self a $500 AWS charge and then deciding they cloud services were dumb.
I don't know. I just tried to frame the problem from a customer's point of view, because cloud vendors' statement that customers would not like a limit is (IMHO) limited by their POV. Customers do want a limit, but not the way that cloud vendors would implement it. I think a huge part of the problem is understanding what exactly it is that you need when you use a cloud service. (This is varying from customer to customer, and from service to service, of course. You usually have important services that must be running, and others where an outage would be unpleasant but not critical.)
> Nobody wants to spend any money really right? But you need to spend some to avoid losing service. That’s just a cost you don’t like but need to pay.
That is not the issue. From a customer's POV, I would be ready to spend extra to keep the service running, but there is a limit where I'd prefer an outage because I can't bear that much. There are two problems with that: First, the limit is blurry. Second, a simple hard limit would leave me with a huge bill AND an outage. I would want to be able to choose one of those evils, not be left with both. And these two problems compound.
> But I don’t know much about this at all. My whole experience was accidentally getting my own personal self a $500 AWS charge and then deciding they cloud services were dumb.
I don't think they are dumber than the alternative. If you run your own hardware, you have a hard limit in both cost and computing power. You could technically get that with the cloud too, but it is not usually offered because it doesn't really solve the problem, but neither does it for for your own hardware.
That said, it would be nice if the major clouds would offer a "hard limit" option, but it really only works for "unimportant" applications that are cost-sensitive and can take an outage.
Of course cloud vendors do put themselves in the customer's position, and that's why they say that customers would not be happy with a limit, even though they are asking for it.
But to extend that to the billing case, you’d have to have a partnership with your customers, not just a dashboard where they push buttons and an API where you add or delete machines.
Maybe the website goes read only except for admin traffic when the budget is exceeded, for instance. Not as a bespoke process each company has to reinvent, but as functionality provided by the vendor.
It's different here, sure, but the providers optimize for not letting customers shoot themselves in the foot, and remediation via bill forgiveness is a fine solution -- from the provider POV.
A company isn't liable if their customer gets themselves hacked because they decided to not use any of the many MFA options available to them and neither is a company liable if the customer set a billing limit rule that they executed correctly.
Companies can simply not be trusted to tell the difference between a foot-gun and a..whatever a good kind of gun would be...
> aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | xargs -n1 aws ec2 stop-instances --instance-ids
Will stop all ec2 instances.
The real fix is scoping credentials on aws - if you don’t use an account or role with limited permissions then even if they had this toggle the first step in an attack would be to disable this option.
Reversibly (i.e. shut down compute, don't delete anything, allow the customer to review, fix and reinstate quickly) terminating service is a minor annoyance for hobby/experimental setups, and in those, it's much more preferable than having to open a support ticket to deal with a massive bill.
Having quotas that the customer can increase themselves (but has to manually choose to increase) on storage prevents storage related surprise bills, and the rest you can shut down (optionally, make the user choose up front what they would prefer).
What am I missing? Too many commercial customers picking "experimental" initially and forgetting to change it?
I would also expect a price cap feature to be useful for experimental/no-approval-required projects at work. In fact, if I ran a cloud project for work as a small team-internal project, a cost explosion would become an even bigger bureaucratic nightmare than if it happened at home.
The problem is 100% technical. Detecting unexpected charges, scaling and restriction in real time is hard.
It's easier to just charge people money than come up with good ways to avoid charging them, and deal with edge cases as a manual process.
Sure. I get it. What company has an internal team that's like "ooo... lets find ways to cap the amount of money people pay us".
No one.
That's why.
> there are reasons other than malice and avarice
Right.
It's just avarice. There's no other reason.
> It's just avarice. There's no other reason.
The problem is technically solvable, it’s good for customers.
All you need to do is prioritise actually doing it.
In what stupid world do we pretend this is somehow good for consumers? Racking up thousands of dollars in bills?
It’s just the status quo.
They’re not not fixing it because it’s hard; that’s just the convenient excuse; they’re not fixing it because they don’t want to.
We have been measuring CPU, MEM with extreme granularity, how about considering price as a resource and measuring the same way, so that a service with a price cap can self manage and self terminate according to some priority field?
This might not be the actual solution, but we have been at this for a very long time, seems like there is not even a hint of an attempt at solving it by the giants. This is about incentives, sorry.
It made me feel much more comfortable hacking around and not needing to worry that I'd accidentally create a render loop or something that could rack up a bill whilst I wasn't looking
This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.
If you look at lots of these threads you’ll see that many people don’t want to provide phone numbers, lots of people ignore emails, even repeated ones, directly to them, from billing.
This isn’t a technical problem, it’s a service problem. I can see the hn posts already “my site went viral and <HOST> shut me down”
When you're paying $50,000/month for something that generates $200,000/month in value, or if an outage can generate $100,000/month in costs, or if the people that can fix an outage cost $100,000/year, then it's not.
That eventually will be factor into the price like credit card fraud insurance. Better have it be more transparent.
All of these incur costs. How hard a cap do you want?
I'm not talking it down. Maybe people are right about this. We'll see.
- a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out”
- a hard limit which fails until more tokens trickle back in, without shutting down service
I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing.
Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought
I am never going to want to spend 200k of my personal money on some project on a cloud. Never. I don't even want my ant-based basket viewing project simulator to cost me 1 thousand dollars because it went viral and all clouds overcharge for bandwith.
Just let me put in a limit.
E.g. when my limit is reached to they remove the database, along with all backups, and all objects on S3? Since storage is billed, it should also be stopped when the limit is reached, right?
I think in practice among companies paying most of their revenue, there’d be zero interest in this, while it would be a lot of effort to implement.
This is really something specific to hobby projects, which just shouldn’t be using those “unlimited potential cost” services.
My tinfoil hat is that a lot of cloud billing is accidental, probably from "lab environments", and they don't want to provide a way to budget/limit these.
“Turning them off” means deleting them.
The provider could reject further access to them (reads / writes) once the limit is reached. The cost of actually keeping objects as "cold" storage has a natural cap per billing cycle since those are billed based on time.
Exactly - the whole point of a cloud provider is scalability. If you're doing a personal hobby project, get off big scalable clouds and get yourself one (or multiple!) fixed-price VPS or dedicated servers.
But as I think of it, I think what people really want, for hobby projects, is not so much the scalability, but the managed offerings. They want zero-ops, zero-maintenance, zero-server-updates hosting, with a fixed price and hard limits. It won't be infinitely scalable, but it doesn't need to be - it's a hobby project.
They just don't wanna sysadmin a server of their own. Which is completely understandable.
There's room in the market for something like this.
offerings were published on the main page and afaik no takers. We migrated off whatever hypervisor we were using onto wok/kimchi and finally to proxmox, so my scripts still work, but proxmox has turnkey linux "quickstart" servers now as well as lxd, so there's less reason to use my scripts.
I only know of Hertzner.
For a lot of the personal projects and early stage startups that are most terrified of these kinds of mistakes and therefore avoid autoscaling products like fly.io, we sincerely would rather have the entire account shut down when it goes over budget than ever see a bill that's higher than our net worth. A line item veto somewhat alleviates that concern but not fully.
They indicate at the end that they're going to do something along these lines, but what they're describing there also seems over-engineered compared to a simple circuit breaker that kills the account or some subset of it. Is there a good reason for these providers to avoid implementing that feature, which on a naive look seems far simpler than what they've actually proposed?
> I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal.
"Terminating service is a big deal" how? I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend?
No one is asking for automatic circuit breakers applied to all customers indiscriminately, but I'm not seeing any justification in here for why an opt-in circuit breaker is technically or legally challenging to implement.
As someone also involved in billing systems for public clouds: in theory there's no difference, but in practice there is a world of difference. This is the sort of situation where the end user is commonly surprised with the consequences of their own decisions. At MGC we have some "soft shut down" processes, and we constantly hear stuff like "I know I said shut down, but this is the one situation where that really didn't make sense"; where examples are "storage which keeps backups became unavailable", "a very simple but critical user auth system disappeared", "I had no idea this was still running on my account", or "OMG not in the middle of the weekend", etc. You can build heuristics and tracking into the system to minimize these situations, but that's a lot of work.
So yeah, it is a valid use case and something many CSPs would like to provide, but implementing something that is actually better than nothing is non-trivial.
My implementation of that feature would be when you turned it on fly would just kick you off.
No serious operator wants a provider to turn their system off. So if you want that it’s pretty clear you are hosting a silly system. Which likely costs more to support and drives less margin.
but you got my attention, i can try fly.io
If you're really only looking to spend $50, we should put our cards on table and say that we're generally not making product pricing decisions with you in mind. If your needs are pretty straightforward, there are hosting providers that will do a better job of serving that business than we will.
IE, you clearly don't want to terminate or shut down an account if they get too close to a cap. But what about things like a warning email, service slowdown, ect?
Likewise, the old "slashdotted" or "hug of death" might be an appropriate result when something goes beyond a reasonable safety buffer?
Anyway, just curious. It's clear that it's a complicated topic, and the real constraints and challenges are interesting.
If there was a way to make caps work for our core customers, we'd do it. We're open to ideas. A theme of our work this past month and these next several months is extracting maximal value from ANFWWAONW, our new billing system. The thing you have to remember though is that our belief about our core customer is that they are averse to nothing more fiercely than service disruption.
We're not in principle opposed to caps. We just don't have a product story for them that we're comfortable with. Keeping you from spending more money than you wanted to is an explicit product goal of ours (again: see post); we're just very wary of trading availability off against that goal.
I already automate apps, machines, etc with the machines API and GraphQL, so my big worries in this area are:
- Woops, some bad logic deployed too many machines (sounds like this policy helps) - Some kind of mistake or attack that just explodes bandwidth usage suddenly
Of course that comes up again when anyone sends a request, but that feels sort of in the same category.
That said, I do understand that building your service for people like me that’d rather be restricted to just $50/month doesn’t really make sense.
Of course it’s nicer if you can be certain that your provider is going to refund you the excess, but I feel like it’s hard to count on it. Or at least, harder than having explicit rules, which you just can’t really do for those sitations that are sensitive to fraud.
Honestly, if I did set a cap I’d be very much aware of the fact my app could suddenly die in a situation where my deploy token were stolen (but it wouldn’t matter for me, since it’s a hobby project, I care about controlling costs, not uptime).
This sounds more and more like an insurance policy; as opposed to a "sudden spike of load because of popularity" situation.
IE, all our talk about usage caps is really missing the point about what needs to be protected against.
Some people are going to want you to go to 1Mb/s if you blow your bandwidth limits, or cash spend. Some might want you to go to 10 connections at once. Some might want you to just disable networking entirely.
And what happens to your storage when you blow your billing? Or CPU time?
It all becomes a hairy mess of state machines and companies wanting precisely _their_ requirements met, so you try to offer as much as you need to still have a compelling offering that enough people want to use, and no more.
Probably at best you could provide an API to make it easy for customers to build their own state machine, but that's fraught because then the customer will still blame you even if their own code did the wrong thing.
Now evaluate your “cap” rules (which will be just as complex) and feed that back to the actual admin/control plane of the service.
The actual problem that people want solved is "the customer wants predictable, budgetable upper bound periodic cost". You are not unique in offering a service where this is a desirable property. Realigning this sort of cost structure is the bread and butter of insurance industry, and no, as much as they'd like to, they don't actually do it by making sure to stop the earthquake before it knocks down more of your house than your price cap.
i think is what gave that vibe off. I was on a read-only phone in bed and saw the quoted message. got up and logged into the PC to think about what to say. It may be time for cloud providers to dissuade small users away, instead.
Major cloud-style companies dont drive significant revenue from your $50/mo cohort. And a $5/mo dev account is basically courtesy for the sales pipeline. The vast vast majority of revenue is “enterprise” sales with private pricing and spend in the hundreds of thousands to millions range.
I get that the underlying issue is that your target consumer is whales who eat orders-of-magnitude pricing spreads as normal opex, and that anyone who comes in with a budget is barely a consideration. It's still absurd to pretend that pricing is too hard. Like, I'm not going to confidently assert "lol you can give up C, it'll be a rounding error anyway" but like, billing cycles are absurdly long on the scale of your technical constraints.
For the nth time on this thread: we don't make our quarterly nut billing people looking to spend $50/mo an occasional extra $1000. The people who actually pay the bills here do not want this feature.
They definitely want Accident Forgiveness, though. Which means it's going to cost us money to do this. And that's fine; they're growing, we're growing.
What I'm saying is that capping billing is not the same thing as shutting down parts of the platform. You're redirecting complaints about the latter by focusing on the difficulty of doing the latter in real time, which is granted but gross overkill.
The former is something that is not only doable, but something that already happens regularly in an informal capacity and nobody believes you don't have the data to price it.
Now if you have some _other_ proposal for how billing and service limits could function Im legitimately interested. But I dont see anything at all specific or actionable in your replies. Insurance is interesting for _some_ facets. Im curious how you think that aligns with dynamic resource utilization and what happens at the boundary.
Billing doesn't have to be so complicated you can't calculate it in less than a minute. That's a technical failure. Surely you can imagine a better way? If you really think it can't be better, then it's hard to argue against "brain rot".
Also on most systems it will work perfectly fine to use an estimate of the price per unit when calculating the bill for the last couple hours.
I'm sorry that "prioritize it" isn't very helpful, but it's true. If the calculation takes longer than that, it's a management-induced problem.
If we're positing a system that can already do that, then you need to be more specific about the problem you're describing because it's not obvious.
This response explains the problem best: https://news.ycombinator.com/item?id=41334596
> I don't think anyone with a serious app running on us will use a cap. Just stay fixated on this scenario: a deploy-only token gets stolen, and the attacker (like most cloud attackers) uses it to stand up a bunch of Monero miners. As a consequence... their main app goes down? Who would be OK with that?
Reading between the lines: If a customer's utilization suddenly spikes, the assumption is that the customer's revenue will follow. IE, if my utilization goes up 10x, my revenue will go up 10x, so I'll happily pay the bill.
What they are providing is more like an insurance policy against hacking.
And that is the answer that I was looking for.
Obviously, cloud providers are capable of billing. It's not unreasonable to expect them to offer, at minimum, "Look, we're not evaluating billing continuously but we might, at our option, try to shut down your services after $X and won't bill you excess of $Y" as a (optional) hard contractual term. Having a larger, well-capitalized party absorb risk on behalf of a smaller party for a fee is a business model humans know how to price. It's just not the desired "product".
Note that this doesn't even per se require any technical artifact to implement, just very primitive metering and lush margins.
You can’t scale to individualized service for $50 per month/quarter/year users, that’s true. But you can shape policy for a demographic with… shall we call it flocking behavior, for lack of a better term?
There should obviously be multiple types of caps. AWS and others have set a precedent that they can get away with "gotchas" for anyone who isn't paying attention.
It's really the primary business model. People that aren't watching costs have more and more sneak in there.
Does anyone know of a service like fly.io that has a perspective that's more friendly to bootstrapped startups?
One interesting thought was trying to model some of this as an actual insurance method. Think of the cases where an adversary of the customer might inflate their costs through 3p usage/traffic. Providers dont want to incentivize those adversaries, deny the customer service, or charge them for unuseful service. Normally it devolved to credit/forgiveness, but then that moves the customers business model risks to the provider. What if this functioned similar to an insurance model; very cheap/baked in forgiveness for everyone (as today), then based on risk profile (porn/games/polical/gambling, or previous occurrence) the customer gets the option of buying forgiveness insurance or self funding their risk. The real sticking point is around perceived/potential conflict of interest and goodwill for the provider to say “pay us more for a thing that you cant directly control.”
Proxy for popularity: https://trends.stackoverflow.co/?tags=elixir,next.js,django,...
That's why residents in california probably pay 100000x / unit of water than agriculture customers do .
The same works for the cloud. An operation that costs $100000 does not cost the operator $85000 in operations on the margins.
Yes the metering and pricing schedules are necessary, but punitive pricing for accidents really is just an artifact of the system.
I'm guessing the biggest reason AWS provides forgiveness is that if anyone took them to court over a bill, a court would throw out the charge once the wholesale cost was revealed.
> We reserve the right to cut you off.
Presumably there was already a point where you were going to say “you keep messing up with the auto shut-off, at some point we’re going to stop refunding you”
I’ll say it’s nice to have it explicit.
GCP also waived the whole thing - even refunding the $4k they had already pulled from my payment method.
Also, do everything except databases inside kubernetes. Deploy kubernetes across multiple clouds via wireguard. Label your instances properly on each provider. Prefer bare metal instances where available. Migrate your workloads accordingly. Force cloud providers to earn your money. Don’t however have both workloads running at the same time in multiple providers as you will eat insane data costs.
Second paragraph was useful, thanks!
Pool unused "accident" credits, automate the forgiveness request, and sell it to the highest bidder.
Checkout their products
But I'm happy to remind people here of it!
"Hey, SMB? you probably don't need our services. While AWS, GC, etc would be happy to take your money anyhow, [...]" I dunno. obviously any sort of thing like this has to clear all the departments because i imagine it increases support load.
I get that this is a lot of venting about people's issues with cloud providers writ large, but damn.
as i mentioned elsewhere i'm intimately familiar with pretty much every intimate detail of "cloud" from hardware, software, network, cooling, and ops (i wouldn't call me a dev. I don't think anyone else would or should, either.) I've bootstrapped cloud services from empty racks twice and repurposed existing hardware for cloud once.
I understand why there's no "cap" available on any cloud services. I mostly have a problem with capitalism, which is ironic, considering this site.
Yeesh.
We agree about the underlying problem! You don't want to spend $5000 in a month for services you never wanted. We don't want you spending that either. We'd rather just improve our billing so that you can fix this after the fact without trading off availability.
But I have just planted a flag: we are prepared to be wrong about this. I don't think we are, but like, I'm the only one. :)
You know what you can’t fix after the fact of getting a huge bill? a heart attack.
I'm pushing back, and I'm going to keep pushing back; if we do this feature, it is going to be kicking and screaming. :)
I guess I’m slightly confused because I thought one of the nice things about cloud services is that they give you the ability to fit your infrastructure to your size while you are trying things. If I’m still trying things, I might not even know if I’m serious yet, right?
In the end, I've just set up a 5$ vps where I self-host all my apps. That removes all the stress.
To be super honest, you might be right too. You might go down a huge engineering effort to build this in only for 5% of your customers to ever engage it. I think the real question is what percentage of your customers will feel better knowing that the have the choice to set those limits, and how will that comfort actually improve their trust with Fly, and cause them to choose it over another cloud provider.
It may end up being a lot like this feature we had in a platform that I used to support. It had a just-in-time analytics pipeline that at one point required tens of thousands of dollars in compute, storage and network hardware alone to function. Based on our analytics, it was barely used compared to the usage of the rest of our app, which made the zounds of resources and fairly frequent support attention it needed feel silly in comparison, so I advocated to sunset it. Product assured me that, regardless of how silly it might be to continue supporting this feature, it was a dealmaker, and losing it would be a dealbreaker.
So yeah, y'all might be right in that the majority of your customers don't actually want it. But maybe what they do need is to know that it's there, ready for them if they ever need to engage with it.
This is an overlooked issue: billing caps are hard to implement and will likely incur losses for the cloud company that does.
Take an object storage service as an example. Imagine Company X has a hard cap at US$ 1000 but some bug makes their software upload millions of files to a bucket and rack up their bill. Since objects are charged in GB/month they will not reach the cap until some time later that month. Then, when they do, what does termination of service mean? Does the cloud provider destroy every last resource associated with the account the second the hard cap is reached? If they don't, and they still have to store those files somewhere in their infra, then they'll start taking a loss while Company X just says "oops, sorry".
That's what tptacek is talking about: you want to NOT destroy the customers' resources because they can quickly figure out that something went wrong and then adjust while still maintaining service. But the longer you keep the resources the more you're paying out of pocket as a cloud provider. If you can't bill the overages to the customer, which a hard cap would imply, then you're at a loss. Reclaiming every resource associated to an account the moment a cap is reached is an extreme measure no one wants.
A hard cap then becomes only a "soft" cap, a mere suggestion, and cloud providers would then say "you hit the cap, but we had to keep your resources on the books for 12 hours, so here's the supplemental overages charges". Which would lead to probably just as many charges disputes we have today.
Storage fees are a lot less risk, but if you want to cap those then you should cap the number of gigabytes directly. That prevents the overage issues you describe.
Having said that, the reason why I personally would be happier with the ability to set a hard cap is that if I'm going to put a project on fly.io, I'm spending my own personal money on it. If I'm spending my own personal money on it, I want a guarantee that it cannot possibly cost more than a given amount. When it's my own money on the line, I absolutely want the service to shut down rather than have even 0.01% chance of costing me a lot of money.
The moment I'm actually building something for a business, the moment it's company money instead of personal money, then priorities change and everything you're saying makes sense. But as long as I'm just a single developer playing with stuff and billing it to my personal credit card, I want that guarantee that I won't accidentally make myself go broke.
Also, you could turn things off in the inverse order they were turned on until the cap is satisfied. So all the crypto miner instances would be turned off before database backups being deleted.
Should you turn things off until you reach a certain spend rate? Then you can set your cap high enough for your server +1, but no more.
Putting both my personal/private as well as work hats on: I’m not sure I would ever want to enable it.
It would make a lot more sense to throttle service, say 80% instead like ISPs do. Slow bandwidth/fewer cpus would raise flags without necessarily breaking anything.
Like telecom providers of course cloud providers could have metered service billed under "get only as much as you paid" policy but it's obviously they totally and in bad faith don't want that.
Couple that with the fact you can achieve quite a bit on simple set-ups that are adequately sized to begin with, you can save quite a bit.
Not all of us need elasticity, or environments being spun up/down on commit.
But then you run into the risk that the client will never increase those hard limits and ... pay less than they could. Not good for business.
This^ Don't feel bad about asking for credits when you accidentally make a costly mistake.
I had a $600 surprise water bill. It was (partially) forgiven because the water department could drive to my house and see evidence of the leak next to my water meter. It did turn out to be on my side of the meter, so it is my responsibility.
If the water department had driven to my house and seen evidence of commercial agriculture (so to speak), then it would not have been forgiven.
---
The parallel here is that the water department can't come into my house uninvited - the cloud provider SHOULD NOT have intimate access to the running code, but they are able to observe some patterns without 'breaking in'.
---
Side note: the size of the bill and the amount of forgiveness was largely driven by waiving an 'excess usage' surcharge - similar to how you can get a discount for cloud service reservations.
Whenever I've had to water new grass seed I've always been surprised by my water bill.
[edit]
14, not 12HCF.
I've wanted to put it on solar the entire time i've had it but the start current is 18A and an inverter that can handle that is (or maybe was, idk) real expensive, considering.
anyhow if you saw the area around my house, in the dog days of summer, right now, you'd think it had been raining non-stop all summer - in fact, it hasn't really rained at all since the beginning of june when we had 12" - i've personally watered 6+" over about 1/3rd of an acre, so like 50,000 gallons since then.
edit: oh crap i forgot the reason i wanted to comment on this at all! My neighbor recently had some issues with a pipe running along the bottom of his house that sprung a leak and they got a surprise $600 water bill - no forgiveness. the water company did, in fact, have the audacity to tell my neighbor "there's a light on the meter if you have a leak." His meter is like a quarter mile from his house, first of all. and second of all, when they read the meter and saw the light, why didn't they go across the street and knock on his door, or make a note to call/mail?
I'll tell you why. $600
Speaking as someone who has had to run a gasoline generator to pump well water and is about to move it to solar, with great rejoicing.
https://eg4electronics.com/wp-content/uploads/2024/04/EG4-18...
Max output is 50A, so it should be plenty for your pump and mine. You can also chain them together if you want huge battery capacity.
!! oh wow, they actually support the full open-circuit voltage that my panels can be series linked* to reach! the only thing i saw many years ago was like... the power wall!
Not only that, it can run every panel i currently own all by itself. that is a leap forward, compared to the last time i checked (a decade ago or so, it was depressing to want a separate solar grid, you need the grid tie stuff so you can run your house if the power goes out, but i mostly wanted the solar to manage water and lights and if possible, the small window/wall aircons that i use to keep servers cool.
Run on the cheapest VPS provider with human support.
Make a quick and easy switch if they get too shifty.
As mentioned in the post: the hosting providers don't actually pay marginal costs for transient mistakes, so neither should honest customers.
Given that the very teleology of cloud computing is digital & ephemeral resources, having an actual human face associated with it makes it tangible in a way that is hard to place.
My little brain can briefly understand complex computer systems on a need-to-know basis, with knowledge constantly coming and going based on the demands of the day. I can never fully "understand" my cloud infrastructure. But hey, Kurt's standing there. I like Kurt. I can understand Kurt as he's a human and so am I and he's in the same exact place I am. Let's work hard and make cool stuff, what do ya say Kurt?
Uh, what is the magnitude of these water bills? I’m fairly certain I can predict it to within a few dollars because it’s only $35 in the first place.
Also, I get receipts by postal mail even if I auto pay them, so I guess that makes it easier.
What would it take for providers to listen to real customers here?
I have $25k in cloud spend that we absolutely cannot go one single cent over due to the politics of internal budgeting. That's my reality. If you want my $25k, I need to ensure that I don't spend more than this amount.
As is, my solution is to use old-school pre-rented, long-term contracted commitment VM hosting providers. This is really the only way to guarantee that you are paying an exact amount and no more.
But, I really would like to use a more scalable system that didn't require pre-provisioning. And, I wish people would believe the customer when they say something and not continue to gaslight us.
Providers say it's impossible, but I don't see how it would be so hard. Here's my sketch of how it could work:
The main component is a system that monitors billing events and watches for the slope of the bill to ensure that there is enough runway to stay under the cap. Optionally, they could implement rate limits on resource creation to ensure that a sudden surge doesn't outrun the monitoring.
You also need notifications for when the projected spend exceeds the cap. Optionally, you could implement a soft cap, where no new resources can be created.
And finally, you need the hard cap where things start to get deleted. If they're feeling generous, the provider could implement a period where VM's/lambda's/etc are shut off, blob storage is not accessible, and so on so that the account holder has some time to fund the account and/or fix whatever is causing the overage.
That set of features are all totally within the competency of a cloud provider. Knowing how much things cost, billing for them, and turning them on and off is their main business. And I can't believe that they expect us to believe that it's impossible to do that tracking.
This is how I do system migrations. There are escalating warnings until one day the service is shutoff for an hour and turned back on. That wakes up any laggards that missed the dozen or so communications over a three-to-six month period. Finally, after a few more days, the service is shutdown for good and then data is deleted a week after that. Though I almost always keep a copy in cold storage. But that isn't necessary as a provider with a limited relationship to the client.
At $25k/mo spend, you can talk to many cloud providers (certainly including us) to work out an "I can't get invoiced for more than $25k" solution which will not involve having your app turned off abruptly when the 2,500,001st cent get spent in October. What you'll notice in this thread is that people generally want billing caps for accounts they plan to spend, like, $10/mo on. And we can build cap systems for those people --- but they'll involve turning parts of the platform off for them.
I'm really pleading with people with strong opinions about hard caps to do the exercise of working out how these billing systems work. There are a huge number of apps running here, across a huge fleet of physicals, running in almost 40 regions around the world. Each of those apps has several different kinds of resources that meter at different granularities and incur different costs. Speaking as a witness to the creation of a new billing system just a month or two ago: it is kind of a miracle that these things work at all. Do the thought experiment, read some Call Me Maybe posts, and then tell me it's obvious that this feature should be straightforward to build.
But my mistake that aside, I appreciate your reply. I saw in another comment thread that you added that you have a blog post in the pipeline on building the new billing system. So I'm really looking forward to reading that. I've enjoyed reading other billing/payments content. And I'm sure your post will also be insightful and highly detailed.
I think it's so so fascinating that this feature is consistently solved at the contract/legal and customer support "layer" of the stack. That's really unpleasant to me, because it is a lot harder for me to wrap my head around the specifics of how different edge cases will play out.
Like as a programmer, I've built up all these skills on reaching technical documentation, understanding systems, their limits, and complex interactions. But instead of using that muscle memory, I have to try to talk with a human and deal with the seemingly intentional vagueness of the legal system.
It feels to me a lot like the story from Mitchell Hashimoto about dealing with the bank for his startup, where he was dodging calls from his account executive and generally behaving in a way that the bank is not used to from enterprise clients. [0]
I'm ready to admit my behavior is anti-social and irrational here. But, it is what comes natural to me.
This is meandering now. But, I just want to sneak in a bit more info on my use cases, since you also mentioned that people are wearing you down and you just might implement this if forced to.
I build and run internal tools (think CRUD & reporting/analytics) for a small department in an extremely large enterprise. Our stuff is on the order of 99.9% available. So not particularly great but not not terrible. But, others are extremely bad. For example, one vendor has over 36 hours of scheduled downtime per month. And that system is way more critical to the business than mine.
So the standard that my coworkers in the department have come to expect is very low. If the tool is down, they just continue with their day doing some other task.
Many of the systems I manage are also purely background jobs. And no one would even notice if they were down for 12-24 hours.
Lastly, we have external backups for everything (on a different provider) and every system's deployment is automated from creating the VM's, networks, and block storage all the way through to installing system dependencies, the app, and data.
So, if a system were to magically get deleted some day, I'd get paged and have it back up in about an hour. And this is totally fine for our business.
On the other side though, there will be dire consequences for my career if we go over $25k annual spend. Even if the bill arrives and we have to contact support, it will give my management a heart attack and they will absolutely remember come review time.
Given this environment, I'd really appreciate the ability to protect myself against misconfiguration or leaked keys causing me to get possibly fired. The data will be fine. And systems can be restored quickly. But the damage to my reputation, compensation, and future job can't be restored quickly.
[0] https://mitchellh.com/writing/my-startup-banking-story
> Many of the systems I manage are also purely background jobs. And no one would even notice if they were down for 12-24 hours.