We run plenty of our own models and hardware, so I get wanting to have control over the metal. I'm just trying to figure out who this is targeted at.
- having the GPU compute in the same data center or at least from the same cloud provider can be a huge plus
- it's not that rare for various providers we have tried out to run out of available A100 GPUs, even with large providers we had issues like that multiple times (less an issue if you aren't locked to specific regions)
- not all providers provide a usable scale down to zero "on demand" model, idk. how well it works with fly long term but that could be another point
- race-to-zero startups have the tendency to not last, it's kind by design from a 100 of them just a very few survive
- if you are already on fly and write a non-public tech demo which just gets evaluated a few times their GPU offering can act like a default don't think much about it solution (through you using e.g. Huggingface services would be often more likely)
- A lot of companies can't run their own hardware for various reasons, at best they can rent a rack in another Datacenter but for small use use-cases this isn't always worth it. Similar there are use cases which do might A100s but only run them rarely (e.g. on weekly analytics data). Potentially less then 1h/w in which case race-to-zero pricing might not look interesting at all
To sum up I think there are many small reasons why some companies, not just startups, might have interest in fly GPUs, especially if they are already on fly. But there is no single "that's why" argument, especially if you are already deploying to another cloud.
But none of this answers my question.
I'm trying to understand the intersection of things like "people who need GPU compute" and "people who need to scale down to zero".
This can't be a very big market.