Why does memory and other hardware tech always seem to magically come to fruition? At least compared to software
Is it because they're more conservative in their announcements? More certainty in a path forward? More $$$?
https://en.wikipedia.org/wiki/Bubble_memory
And then it just basically turned out to be a flop.
The neural net software algorithms have been around for decades. What made LLM’s feasible are the hardware advances to achieve unprecedented scale, just barely providing the ability (at great cost) to train today’s LLM models. Transformer architecture might be called a software innovation, but RWKV Raven gets similar performance to transformers and is built on decades-old RNN technology. So it is the hardware that was far more instrumental than the software in achieving LLM’s.
Counter to that argument: had google not done neural net research for google translate and proved the transformer approach scaled and performed well in their “attention is all you need” paper, people wouldn’t have spent the money to train foundation LLM models and we would not be having this discussion, so the software really mattered more than the hardware.
In reality I think it’s a little bit of both.
Even while they missed the opportunity, Gpt-1 was released in 2018 by OpenAI.
Then they incrementally added parameters in the next versions.
Most software (and hardware) is not like this.
See also memristors, that HP in particular have been saying is coming in the next couple of years, fundamentally changing the entire computing industry, for decades.
FWIW, gallium arsenide was also supposed to take over from silicon for CPUs once speeds went above 25-MHz (or so). A senior colleague of mine said, "There are a lot of smart people whose kids's college tuition depend on making silicon a little better every year." And so, it came to pass. GaAs is definitely important but we've got multi-GHz silicon CPUs now.
[1] https://www.microsoft.com/en-us/research/project/hsd/ "Project HSD: Holographic Storage Device for the Cloud"
At what point does an influencial government say "it's unlikely there is substantial innovation to be uncovered for ___ product" and begin the process of deprivatisation for good of the public?