Like, right now especially in the US you’d probably sound crazy saying that one day nobody’s going to buy a gasoline car, and that EVs are going to completely replace them. But that’s basically an inevitability based on the direction we know technology is going to go, it’s only a matter of when.
The point being that the way we heavily rely on LLMs specifically right now is likely to cease to be the case, and that in the future they’ll be almost like an irrelevant component.
You should check the code of both world models (JEPA class for example) and compare to GPT. Many of the tricks stay the same, representation is still embeddings, there is a loss function, etc.
The exact architecture will change, but unless there's a new discovery in that area, we've cracked the text component already. We're hitting the limits of LLMs because of the intrinsic limits of text as a medium. But the way we work with text is pretty much settled, fundamentally.