In a seemingly coordinated nature, large corporations like Microsoft, Facebook and Google all released their LLMs and AI chatbots. Then nearly immediately after open-source versions came out, many in fact.
So what happened?
Were they always there and just waiting for the greenlight?
Or is it just a case of a 4-minute mile?
Google engineers actually created the technology (transformers[1]) which enabled these LLMs, but OpenAI simply released first.
[1] https://en.wikipedia.org/wiki/Transformer_(machine_learning_...
There's transparent evidence that transformer-based LLMs existed before ChatGPT or Llama. There was originally GPT-2, which was not a public release but did get reverse engineered into GPT-Neo and GPT-J (among others). Besides that, there was BERT and T5 from Google, Longformers, XLNet, and dozens of other LLMs that predate the modern ones.
> So what happened?
OpenAI reduced the friction of using AI to a Google login. Other models existed, but were largely not perceived as powerful enough to compete with a hosted solution. The combination of GPT-3's power and relatively-cheap usage fees is what drove it's adoption. Local AI solutions were being developed before OpenAI stepped into the arena, and are continuing to get developed. Many FAANG companies are heavily invested in the openness of AI architectures.
Transformers were first published about in "Attention is all you need" in 2017 by google https://arxiv.org/abs/1706.03762
> There was originally GPT-2, which was not a public release but did get reverse engineered into GPT-Neo and GPT-J (among others).
GPT 2 had a proper release in 2019 with the code, structure, checkpoints, etc. - https://openai.com/research/gpt-2-1-5b-release
> Besides that, there was BERT and T5 from Google, Longformers, XLNet, and dozens of other LLMs that predate the modern ones.
Yes all the big players had various research models. We simply crossed the line into large-scale usefulness just recently (outside of classification/summerization), and like you said OpenAI was the first to really make it accessible to the public.
If chatgpt can steal eyeballs from google or facebook (something that Bing has tried with no success for the last 10 years), it is existential treat to google or facebook, so they must react.
The same thing would have happen with VR. As soon as some company will succeed with VR, your would suddenly see a lot of VR companies.
That may well be true, but both Google and Facebook have been writing Open source AI software longer than OpenAI has been publishing papers. I find it hard to believe that everyone else is playing catch-up with OpenAI right now. I don't think a product like ChatGPT could even exist unless it was massively subsidized.
For a while it was largely academic and not crazy useful (GPT2ish era).
After this they had internal LLM's that could potentially be used to make products. This is where the companies split in their approaches. Google was overly cautious from what I've heard, they were concerned about "safety". Then openai released GPT3.5 and soon after GPT4. During this time Meta took the approach of just releasing their research model outright without first building it into a product. Google then scrambled to build products from their LLMs.
tl;dr - everyone of those are big players in the ML/AI space and had internally trained models for their research. We just recently crossed the line where these models became useful thus this scramble to release products (or the models themself).
EDIT: It's worth noting that the big breakthrough paper about transformers (Attention is All you Need) was released in 2017. People started scaling up the parameter count, optimizing training time, inference cost, etc. since then.
People get a really great new idea, all the smart people jump on it, the field is burned out in six months.
This revolution in science
https://en.wikipedia.org/wiki/Renormalization_group
didn't last long enough to finish grad school.
That idea of "build a language model by predicting the next token" was floating around for a while, I was in two startups that were trying similar things long before it was really feasible.
First we were trying to make something that could write fake abstracts for clinical reports from PubMed as a proxy for clinical notes with LSTM, GRUs and such.
Later we were training CNNs to complete a battery of classification tasks and believed this could make a "foundation model" that could be trained to new tasks easily.
I had a long list of serious problems that doomed many lines of development related to what I was doing and when BERT came it was clear to me that subword tokens were essential (e.g. word vectors are doomed because half the time the most important word isn't in the dictionary!) and also that attention addressed the problem nobody really wanted to face with LSTM which is that in a story about a person you have to copy that person's name and variations over and over again.
It wasn't clear to me that transformers were going to be as successful as they were.
It helps also that there was a big investment in training technology. Often technological revolutions happen when there is a surplus in the base technologies. For instance, experts in 1970 anticipated we were going to get something like the WWW in 1980 but it was really 15 years late because there was no single company that was going to make the end-to-end investment to bring it to the mass market, excerpt in France
https://en.wikipedia.org/wiki/Minitel
(The TRS-80 Color Computer wasn't really intended to be a computer to teach kids BASIC, it was intended to be a terminal for teleagriculture!)
When the web came out, however, the cost of investment in making it work had dropped so low that it didn't take one company to invest in an optimized end-to-end system, rather a bunch of little investments (like the one people made in a 32-bit PC) added up to a network that looked like the computer on the bridge of the Enterprise.