Yes, short term this is right. But at some point PyTorch will have a model.toVHDL() method, and we'll have a PCBWAY-style website for tapeout of the circuit. Nvidia's future looks less bright than they think and their GPU market will certainly pop.
And don't take it too literally, VHDL could be replaced by other hardware design languages, maybe even at lower abstraction levels.
And a GPU is already pretty optimized for inference, no? Like isn't it a bunch of FP mults? I don't think HDLs do well with that, either.
If such AI models will deliver on their qualitative promises, and just the huge cost is the burden to overcome, custom ASIC might be a part of the solution.
If, on the other hand, AI models will still be unsuitable for many applications because of their qualitative issues, it is a much harder and different problem to solve - in this case, the AI bubble will plausibly burst.