With all due respect to LeCun, not him nor anyone else in the field predicted the new emergent capabilities brought within the last few years.
So, he’s saying this is not going to keep happening?
What level of confidence is he putting on not seeing it last time, but being right this time?
Turing predicted it would happen around the year 2000 and take ~1gb of ram.
> I believe that in about fifty years’ time it will be possible to programme computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent, chance of making the right identification after five minutes of questioning. https://academic.oup.com/mind/article/LIX/236/433/986238
I think Turing was right. If I run TinyLlama 1.1B on my computer I can have a conversation where it pretends to be a person. It's small and fast enough that it'd probably run fine on a high end workstation from 2000. If the tech was possible back then, then it probably existed. Keep in mind Turing was the sort of person whose work at Bletchley Park took 30 years to declassify.
Turing was right, but he believed in human intelligence, so thinking linear.
Unfortunately, neural networks appear harder then sought, because Forward-Forward learning with which working human brain, is now considered too computationally hard to be used practically.
Reverse-Propagation neural networks are much easier computationally to achieve, but are magnitudes less dense than FF, that's why we have so much delay - just need thousand times more performance, this is about 30 additional years.
PS I tried Llama 30b on my computer (honesty, tried all Lllama models could fit in 64G RAM, from 13B and more). It is not sprinter fast, but I'm very impressed, on how deep it could think. I must admit, I don't want to talk to many humans anymore, as It looks smarter.
Besides, in his analysis, le Cun consistently spoke of GPT as “writing help, no more no, less”.
It’s the convergence of end of ad economy/attention supply growth and Microsoft’s money forcing AI as the next growth narrative that’s forcing google to join the party - at massive cost to them.
Also regarding coding, they can definitely produce junior level code, especially if you follow a similar trial and error way of thinking which happens all the time during your work, if you try to use an unknown interface.
I do think that it’s quite a leap to assume that. It is very good at mimicking, but the same way some parrots can talk, they are endlessly far from human cognition. That jump is very very non-trivial.
And at least part of that was the (implied, natural, understandable) "linear extrapolation" of the total cost of compute applied - if your intuitive expectation of progress if effectively "oh, what would happen next year if we'd triple the budget" but actually the applied compute power increases hundredfold, then obviously the outcome arrives faster than you expected.
That's made them effectively immune to copyright enforcement so far; I'd go as far as saying the main function of llm/diffusion models is obfuscation of copyright breaches.
But what about for tasks where datasets don’t really exist? I do a lot of PCB design and it’s extremely time consuming. But it’s a niche field compared to text and images. No dataset exists that says “these were the engineering requirements of this PCB and this is the result and by the way the board actually worked”.
So how will we train AI systems to replace a human doing PCB design? It’s probably going to need to learn PCB design from first principles (along with massive help from large transformers when possible, like collections of chip datasheets). Even then, understanding PDF datasheets is something these big companies haven’t really pulled off yet, though I suspect in 5 years that will change.
But my point is that there must be loads of tasks, even on the computer, for which suitable datasets don’t exist and it would be infeasible to create them. Another big thing I do is machine design and again it’s not about designing one mechanical part it’s about pulling in the right parts from all over the world and assuming certain manufacturing processes, and then knowing those processes and then designing all the parts and the assembly. There’s so many different pieces of knowledge in there that are not captured on text or images on the web and would be hard to encode in to datasets.
At some point we’re going to need machine systems that learn the way people do, and that’s going to take a long time to figure out. That’s what LeCun is saying.
One side project I'd love to work on one day would be gathering a dataset of Factorio maps designed to help train a floor planner. The requirements aren't the same as for making a PCB, but they're similar-ish.
That remains to be seen. You wouldn't say a human lacked human-level intelligence just because they had only read every book in The Pile, rather than every book in the Library of Congress.
Multimodal (image/video, audio) is barely being scratched. You likely could 100x training compute and not run out of video content.
Not to mention synthetic data from simulators, compilers, and other sources of infinite ground truth data.
If you listen to the big labs (OpenAI, FAIR, etc) they have started to say they are not worried about data anymore.
Quite obviously, at some point you put the agents out into the real world and they can collect new training data there too; this is what all the ChatGPT plugins are allowing OpenAI to do. The plugins that give good responses which are valuable will be disproportionately popular and therefore overrepresented in the new training set of user interactions.
The point is why should we take that as true when people didn’t have a clue LLMs would take us this far, until they did.
Btw there are lots of ideas on how to deal with the data issue. Multiple directions in research are promising to work around that.