https://en.wikipedia.org/wiki/Universal_approximation_theore...
This theorem explain the limits, putting it in simple terms, most architectures are universal approximators that are constrained by the inductive bias that we give them, so far the approximator arquitectured that is less constrained by the inductive bias is the transformer, so it should be able to approximate any mathematical function, the current problem is that the attention mechanism have a quadratic scaling, so while is easy to scale it in text, is pretty hard to scale it with anything else to the same performance, even if not further discoveries are made, just with the computer power of the future it should be able to scale in every field, even with the techniques of today it gives pretty good performance in a lot of tasks.
This review of the paper an image is worth 16x16 words by Yannic Kilcher explains it better if you are interested.
To put it the form of a rhetorical question: many of these models are public, so why "wait" when you could do the research yourself?
> To know the limits turns their application from hype into engineering.
It would be helpful to know how the models actually work under the hood.
But we made very good use of metals for thousands of years before we understood things like atoms, chemical bonds, lattices, etc.
Some engineering disciplines can be made up largely of empirical knowledge.
Engineering to me is "make the things we want out of the things we have", and not necessarily "design based on complete scientific theories".
The business of these new LLMs is next token prediction with context. This is also now a mission because it clearly works to some large extent. Where most would not have been willing to take a leap of faith prior, many can see some path now. I've been able to suspend my disbelief around language-as-computation long enough to discover new options.