So if the two are somehow connected, then that could have implications for tuning and fighting overfitting
maybe it'd also be possible to design better non-overfitting SVMs
SVMs with well-tuned kernels and regularization are reasonably resistant to overfitting. The problem is that you can easily end up overfitting the hyperparameters if you're not very careful about how you do performance testing.
Otherwise, they just help us in better understanding Transformers and SVMs.
There have been similar equivalences before, for example:
Linear Transformers Are Secretly Fast Weight Programmers, https://arxiv.org/abs/2102.11174
Or policy gradient methods from reinforcement learning are basically the same as sequence-discriminative training as it was done for speech recognition since many years, however, they come with different tricks, and combining the tricks was helpful.
To put it the form of a rhetorical question: many of these models are public, so why "wait" when you could do the research yourself?
> To know the limits turns their application from hype into engineering.
It would be helpful to know how the models actually work under the hood.
But we made very good use of metals for thousands of years before we understood things like atoms, chemical bonds, lattices, etc.
Some engineering disciplines can be made up largely of empirical knowledge.
Engineering to me is "make the things we want out of the things we have", and not necessarily "design based on complete scientific theories".
https://en.wikipedia.org/wiki/Universal_approximation_theore...
This theorem explain the limits, putting it in simple terms, most architectures are universal approximators that are constrained by the inductive bias that we give them, so far the approximator arquitectured that is less constrained by the inductive bias is the transformer, so it should be able to approximate any mathematical function, the current problem is that the attention mechanism have a quadratic scaling, so while is easy to scale it in text, is pretty hard to scale it with anything else to the same performance, even if not further discoveries are made, just with the computer power of the future it should be able to scale in every field, even with the techniques of today it gives pretty good performance in a lot of tasks.
This review of the paper an image is worth 16x16 words by Yannic Kilcher explains it better if you are interested.
The business of these new LLMs is next token prediction with context. This is also now a mission because it clearly works to some large extent. Where most would not have been willing to take a leap of faith prior, many can see some path now. I've been able to suspend my disbelief around language-as-computation long enough to discover new options.