I do feel that AI has been overhyped a bit for now, but what happens when we scale our electricity and GPU production 10x, 100x, go nuclear etc and can 100x AI models - lets see. Its too early to tell really.
Also the fact that 30B models, while less capable than 300B+ models, are not quite one whole order of magnitude less capable, suggests that all things being equal, capability scales sub-linearly to parameter count. It's even more flagrant with 4B models, honestly. The fact that those are serviceable at all is kind of amazing.
Both factors add up to the hunch that a point of diminishing returns must soon be met, if it hasn't already. But as long as no one asks where all the money went I suppose we can keep going for a while still. Just a few more trillions bro, we're so close.
For example, language might not be baked in, but the software for quickly learning human languages certainly is.
To take a simple example, spiders aren’t taught by their mothers how to spin webs. They do it instinctively. So that means the software for spinning a web is in the spider’s DNA.
Nothing interesting without some fundamental breakthrough IMO. Model/agent providers add another level of "thinking" that uses 10x the energy for 10% gain on benchmarks.