Here's what we're doing:
- Fine-tuned models now boot fast: https://replicate.com/blog/fine-tune-cold-boots
- You can keep models switched on to avoid cold boots: https://replicate.com/docs/deployments
- We've optimized how weights are loaded into GPU memory for some of the models we maintain, and we're going to open this up to all custom models soon.
- We're going to be distributing images as individual files rather than as image layers, which makes pulling images much more efficient.
Although our cold boots do suck, the comparison in this blog post is comparing apples to oranges because Fly machines are much lower level than Replicate models. It is more like a warm boot.
It seems to be using a stopped Fly machine, which has already pulled the Docker image onto a node. When it starts, all it's doing is starting the Docker container. Creating the Fly machine or scaling it up would take much longer.
On Replicate, the models auto-scale on a cluster. The model could be running anywhere in our cluster so we have to pull the image to that node when it starts.
Something funny seems to be going on with the latency too. Our round-trip latency is about 200ms for a similar model. Would be curious to see the methodology, or maybe something was broken on our end.
But we do acknowledge the problem. It's going to get better soon.