Actually I feel that utilization of existing hardware is very poor.
For instance, in LLM land, continuous batching is a thing that just starting to proliferate (at least in open source). Many are still doing training/inference in vanilla pipes in PyTorch eager mode, without even a simple `torch.compile` or aggressive quantization, much less a more advanced ML compiler. Architecture optimizations beyond llama's base architecture (like newer attention schemes used during training) are not really picked up yet.
I can go on and on, and go on about Stable Diffusion as well, but what I am saying is there is tons of unpicked low hanging fruit, that's unexploited because... everyone is in a rush to just publish or deploy something, I guess?
Also, Nvidia's margins are insane. The voltage/clock band they run in is insanely inefficient. I hope that is not sustainable, because sane margins/clocks with no code/hardware advancement would bring costs down a ton.