The "even for inference" thing has turned into a bit of a trap imo.

Data parallel models scaled up for training and then could run on individual chips, but these massive model parallel models require a couple of chips directly linked together even to do inference.

So the idea that a competitor could come in with a simple, cheap inference chip doesn't really work.