Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it.
One of Google's moats is, has been, and will always be the scale required to just get into the search game and the tens of billions of dollars you need to compete in search effectively (and that's before you get to competing with their brand). Microsoft has spent over a hundred billion dollars trying to compete with Google, and there's little evidence anybody else has done better anywhere (Western Europe hasn't done anything in search, there's Baidu out of China, and Yandex out of Russia).
VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale.
Unless someone sees a huge paradigm change for cheaper, consumer accessible GPUs in the near future (Intel? AMD? China?). As it is, Nvidia owns the market and they're part of the moat cost problem.
You don't have to run them locally.
It is in at least 2025. AMD (and Intel, maybe) will have M-Pro-Esque APUs that can run a 70B model at very reasonable speeds.
I am pretty sure Intel is going to rock the VRAM boat on desktops as well. They literally have no market to lose, unlike AMD which infuriatingly still artificially segments their high VRAM cards.
Models of any given quality are declining in size (both number of parameters, and also VRAM required for inference per parameter because quantization methods are improving.)
Raw FLOPs may increase each generation but VRAM becomes a limiting factor. And fast VRAM is expensive.
I do expect to see incremental innovation in reducing the size of foundational models.
for end users, yes. For small companies that want to finetune, evaluate and create derivatives, it reduces the cost by millions.