Hacker News Reader

Comment by paulddraper | Hacker News Reader

> Spot instances are underpriced, as most tenants underestimate their tolerance for preemption or overestimate its likelihood.

> In reality, instance preemption is rare in most instance families

I have often seen the following:

1. The demand for spot instances increases for a period of time.

2. Supply is inelastic so prices quickly rise to the full bid amounts.

3. Most users are unwilling to go hours without any spot instances running, so they use very high bids.

You either have to tolerate hours of "down time" every few months, or pay exorbitant prices during once in a while.

On multiple occasions, I've seen the spot instances priced significantly higher than on-demand.

EDIT: Apparently AWS changed the Spot pricing a few years ago though. My anecdotes are 5 years old.

View on HN

To be fair, its also easier than ever before to use spot instances.

If you set up an EKS/K8s cluster and install Karpenter. You can configure it very easily to use spot instances anytime prices are available for less than on-demand and to use on-demand when spot instances are unavailable or too expensive.

You end up never thinking about it but having full availability.

In practice it means the ceiling on your bill is the on-demand price, but you usually average out much lower.

I suspect as more and more people switch over to this model then the use of spot instances will stay closer to full saturation, with those discounts becoming negligible.

I think this comes with the limitation that your workloads might be terminated without your control?

They have a whole section in their docs about this: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-int...

It's all about how your spot instances can be terminated. Perhaps this could interfere with a workload that might take a long while to finish.

On paper your app should be resilient to these things but in practice it's essentially an unforgiving chaos monkey that will delete servers from your cluster.

If you can handle that, you can definitely save a lot though. Especially web servers where it's expected you'll be responding in a few seconds while maybe your background workers are churning through longer lived tasks. The problem there is you need to already be big enough to have different nodes handling web vs worker traffic. A lot of decently sized apps can operate in a world where maybe you have 2-5 nodes on your cluster and your web and worker pods land on any of them based on Kubernetes' default scheduler.

Afair Karpenter handles that. AWS gives some lead time before taking away spot instances. Karpenter puts a taint on the node and spins up a replacement. The scheduler gently evicts the pods, and nothing is lost.

This does require architecting your services to be shut down tolerant, but that's par for the course. If you're using Kubernetes you've probably already settled on the "[herd, not] cattle, not pets" idea.

> The scheduler gently evicts the pods, and nothing is lost.

If you have heavy duty work being done, it might get kill -9'd after the grace period is surpassed. If you weren't using spot instances you have full control over the grace period.

You can of course handle this too, by breaking up your work in ways where it can be interrupted without losing progress but depending on what you're doing this could be very complicated. Basically the takeaway is you can't blindly start using spot instances, even if your app has been running in Kubernetes successfully for a while.

If you're writing your processing code, you can also write it with capability to stop and resume more efficiently.

> I've seen the spot instances priced significantly higher than on-demand

How is the cap price not on-demand cost? Why would you not just swap to on-demand at the turning point?

there's a cost to engineer something that automatically switches or someone going in and manually changing it. so the spot prices has to be higher than ondemand + switching costs. the new pricing models(a couple years old) though have mostly alleviated this

This shocks me too. When I first had to design a system around spot instances I assumed it caps out at the on-demand price. But in practice that is very much not the case. The spot price can routinely go above on-demand, with all the downsides of spot still being applied to the instance.

It is because many systems are only built to support spot. When running a workload, they might not want to give up availability so they just pay the higher price. To be fair, this is a game that AWS engineers. They essentially want to kick people off the infrastructure to free it up for other uses. So this is a way to signal how badly you don't want to be kicked off. A lot of times after a price goes above on-demand pricing, it will dip down below fairly quickly as workloads all re-adjust. Some companies are willing to play that gamble, that a short period above on-demand is still cheaper in the long run when you average out the invoice at the end of the month.

Because you don't want to be pre-empted.

This is no longer the case. Back in 2018 AWS changed the pricing model so now there's no more bidding.

This means "bidding" more won't protect your capacity, AWS will just take it if they need it.

Prices are now relatively flat, and changing slightly over time by a fraction of a cent per day with a cap at the OnDemand price, while previously there used to be lots of from 0.1x to 10x the on demand price and with swings even multiple times within a day.

What is the reason anyone would pay more for a spot instance than an on-demand instance?

If an instance on-demand is, for the sake of easy discussion, $1.00/hr and the spot rate is usually $0.30/hr, it might make sense to bid $1.25/hr (or $1.26/hr) if you didn’t want to get pre-empted.

Most hours, you’d pay something close to $0.30/hr. Some hours, you’d pay over a $1.00/hr, but you’d save money overall against on-demand.

(This ignores Reserved Instances and Savings Plans.)

This is no longer the case since 2018, see my comment above.

Maybe poorly architected software, where it’s cheaper to increase the value of a variable than have a bunch of API calls that check the current onDemand price and call different APIs depending on which number is smaller.

Interesting. While my data does show this being the case in a handful of pools (1-5 out of 13k pools at any given time), it appears to occur only exceedingly rarely.

Reply on news.ycombinator.com