https://www.youtube.com/watch?v=21EYKqUsPfg
Right now, we keep creating new models with larger and larger weights and throwing hardware at it. His argument is that this is a dead end and ultimately we'll eventually go back to purpose fit algorithms like we've always done in the history of AI development.
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
His problem with LLMs are they aren’t general enough… In fact his essay the bitter lesson was what inspired the current LLM scaling.
If you watched the video, you can even see the part where he talks about how AI will fully replace humans as the next stage of life. How wholesome!
Sure, for business, personal, etc it makes sense to build purpose built but that’s not what these companies are trying to achieve in the end. They want to own a nuclear weapon that outstrips everything else as soon as it exists so they “win”