It is definitely not the MNIST classifier you had trained in 2019.
The difference is that you only train it once and the modern LLM machinery sort of takes care of that with large contexts.
It's great that Jev proved this is a viable product. I'd expect a great many research innovations coming from making this work better/faster/cheaper, and around interfacing modern agents with it.
Anyone fitting MNIST in 2019 was already outdated by several years at the very least. GPT-2 was 2019! We already had zero-shot classifiers then. In fact, the paper for GPT-3 was literally
"Large Language Models are Zero-Shot Reasoners"
These kinds of zero-shot classifiers were already developed and used in-house for many years. They just weren't commercialized as a separate product, because anyone who could use an LLM proper could build layers around it to fulfill any classification task like this.
GPT2 was absolutely unusable as a classifier. Using GPT3 as a classifier cost a few order of magnitude more than what this thing is priced at, and the context window was a few thousand tokens.
> These kinds of zero-shot classifiers were already developed and used in-house for many years
This is like Google's favorite coping mechanism for falling behind at AI. "We had everything inhouse for several years, we didn't release it for $reasons."
> anyone who could use an LLM proper could build layers around it to fulfill any classification task like this
You missed the part where it costs more than two orders of magnitude lower :)
I'm not claiming there are major architectural innovations, but that's not the point. Once you prove there's a market, there's a cambrian explosion of innovations.
There already was a market for this, that's why people continued developing them in house for their own purposes. I trained people for this. It was just silently used at various companies without the loud marketing fanfare. The fact that other groups are able to replicate and beat Jev within days should tell you something. That's not "the new normal". That's an example of a product built on marketing hype alone.
Doesn't mean Dropbox was not innovative.
LLMs better than chatgpt existed at Google two years before chatgpt. Doesn't mean chatgpt was not innovative.
Not all innovation is about groundbreaking research. More often than not, it's figuring out what people will pay for. The research to make it work well often comes after that.
Now all you need is to give it more context along with your query.
There's no scenario where you are training BERT online to give you an answer.