How New Are Yann LeCun's “New” Ideas?
garymarcus.substack.com
garymarcus.substack.com
Marcus seems to be in the right (though his seething saltiness seems to me to dilute his message), and LeCun has done nothing so far but double down on dickishness.
LeCun probably should issue a retraction and an apology, before this really bites him in the arse.
And your comment would be reasonable if you hadn't jumped to the "mental illness" comment, that's a bad way to do discussion.
Luckily in the startup world, ideas are worthless and no one cares if you thought of it first but can't execute.
More on academia vs startups: https://twitter.com/mizzao/status/1505529295157948421
The ironic thing of course is that Yann has not been at the forefront of AI for many many years (and Gary, of course, never has). Facebook's research has failed to rival Google Brain, DeepMind, OpenAI, and groups at top universities.
So to the extent that Yann is copying Gary's opinions, it's because they both converge at a point far behind the leaders in the field. Yann should be much more concerned than Gary about that.
Behind? Why do you say so? If anything, they may both (now) be a bit ahead of the curve. AFAICT, while the idea of neuro-symbolic integration is pretty old (Ron Sun, among others, was talking about it ages ago), the idea is still far from widely pursued by the mainstream thread of AI research.
In either case, it's interesting to finally start to see more weight accumulating behind this particular arrow. But I've long been on record as advocating that neuro-symbolic integration is a critical area of research for AI, so I'm a bit biased.
You present this statement as fact when it is still highly debated. A lot of researchers will claim a popular new approach is a reformulation of experiments they did X years ago. It's usually best to see it as a spectrum where some idea are on the same axis, where one end is "totally different" and the other is "renamed approached".
* https://books.google.com/books?id=n7_DgtoQYlAC&dq=Connection...
* https://link.springer.com/book/10.1007/10719871
* https://www.amazon.com/Integrating-Connectionism-Robust-Comm...
As a neo-luddite myself I'm personally fine with that (as I'm personally fine with us throwing money away at CERN), but there are people who still think that AGI is possible and who also think that reaching AGI is a worthy goal, so those people might not be ok with chasing windmills.
I don't have enough context to take a side, but this is not just a rant.
Beyond their interpersonal disagreements, I do wonder if LeCunn is seeing diminishing marginal returns to deep learning at FB...
>LeCun, 2022: Reinforcement learning will also never be enough for intelligence; Marcus, 2018: “ it is misleading to credit deep reinforcement learning with inducing concept[s] ”
> “I think AI systems need to be able to reason,"; Marcus 2018: “Problems that have less to do with categorization and more to do with commonsense reasoning essentially lie outside the scope of what deep learning is appropriate for, and so far as I can tell, deep learning has little to offer such problems.”
>LeCun, 2022: Today's AI approaches will never lead to true intelligence (reported in the headline, not a verbatim quote); Marcus, 2018: “deep learning must be supplemented by other techniques if we are to reach artificial general intelligence.”
These are LeCun's supposed great transgressions? Vague statements that happen to be vaguely similar to Marcus' vague statements?
Marcus also trots out random tweets to show how supported his position is and one mentions a Marcus paper with 800 citations as being "engaged in the literature". But a paper like Attention is all you need that currently has over 40,000 citations. THAT is a paper the community is engaged with. Not something with less than 1/50th the citations.
This is a joke...
However he does seem to have legitimate complaints about the echo chamber the big names seem to be operating in.
Machine learning researchers optimize “performance” on “tasks”, and while those terms are still tricky to quantify or even define in many cases, they’re a damned sight closer to rigorous, which is why people like Hassabis who get shit done actually talk about them in the lay press, when they deal with the press at all.
We can’t agree when an embryo becomes a fetus becomes a human with anything approaching consensus. We can’t agree which animals “feel pain” or are “self aware”. We can sort of agree how many sign language tokens silverbacks can remember and that dolphins exhibit social behavior.
Let’s keep it to “beats professionals at Go” or “scores such on a Q&A benchmark”, or “draws pictures that people care to publish”, something somehow tethered to reality.
I’ve said it before and I’ll say it again: lots of luck with either of the words “artificial” or “intelligent”, give me a break on both in the same clause.
As AI (in the broadest sense) has developed, we always end up moving the goal posts. Sometimes this is because we genuinely don’t know what is difficult and what is easy due to several billion years of evolution. But some of this is because we know how the system works, and so it can’t be “intelligence”.
I think of it as like a magic trick. When you watch a someone do an illusion well, it’s amazing. They made the coin disappear. It’s real magic! But then you find out all they did was stick in their pocket, or used a piece of elastic, and then “magic” is gone.
Essentially this is partially what the Chinese Room is about. You think the Chinese speaker is real, but then you find out it’s just some schlub executing finite state machine.
This is patently FALSE. You can, however, re-run a given prompt 10+ times, tweaking and nudging it into the direction you know you want, until it produces a seemingly miraculously deep result (by pure chance).
Rinse and repeat a dozen times and you have enough material for a twitter thread or medium post fawning over gpt-3.
It is quite incredible that nothing changed about the architecture in gpt-2 vs gpt-3 (just way more connections), yet it aquired fundamentally new behavior - that if performing arithmetic calculation - despite not having large amounts of training data on the subject. I think this is the type of phenomenon that shows we are quite poor at estimating what these systems will be capable of when scaling up. So acting as if we're sure it won't lead to improvements in AI is as idiotic as claiming that it will. There are far too many people on hacker news that follow this fad of being dismissive of AI, because they make the common mistake of equating cynicism with intelligence.
Honest question: what's "intelligence"-like about Stable Diffusion?
There is some intangible property I observe when I look at a human and determine they are conscious. There is some intangible property I observe when I look at a dog and determine it is conscious. There is some intangible property I observe when I look at stable diffusion and determine it is conscious.
Some attempts to explain this intangible property have been made. Almost all of the time disagreements in these explanations boil down to semantics. Yes, I consider the ability so solve problems a demonstration of intelligence. Yes, I consider to Stable Diffusion to be solving problems in this way. Also yes, I consider a hard-coded process to be behaving in a similar way.
At the end of the day we seem to define consciousness as something that makes us sufficiently sad when we hurt it.
https://www.facebook.com/722677142/posts/pfbid035FWSEPuz8Yqe...
"'[...] Yann LeCun, [...] is on a mission to reposition himself, not just as a deep learning pioneer, but as that guy with new ideas about how to move past deep learning'
First, I'm not 'repositioning myself'. My position paper is in the direct line of things I (and others) have thought about, talked about, and written about for years, if not decades. Gary has merely crashed the party.
My position paper is not at all about 'moving past deep learning'. It's the opposite: using deep learning in new ways, with new DL architectures (JEPAs, latent variable models), and new learning paradigms (energy-based self-supervised learning).
It's not at all about sticking symbol manipulation on top of DL as he suggests in vague terms. It's about seeing reasoning as latent-variable inference based on (hopefully gradient-based) optimization.
Gary claims that my critiques of supervised learning, reinforcement learning, and LLMs (my 'ladders') are critiques of deep learning (his 'ladder'). But they are not. What's missing from SL, RL and LLM are SSL, predictive world models, joint-embedding (non generative) architectures, and latent-variable inference (my rockets). But deep learning is very much the foundation on which everything is built.
In my piece, reasoning is the minimization of an objective with respect to latent variables. If Gary wants to call this 'symbol manipulation' and declare victory, fine. But it's merely a question of vocabulary. It certainly is very much unlike any proposal he has ever made, despite the extreme vagueness of those proposals."
Ie. the whole "idea" being "hey, for gai we need something different than this gtp3" tweet, not "idea" as in "hey I invented this new thing I call LSTM, check it out [link to paper, results what not]".
Well, the LSTM fella does pop up later calling out Lecun for "rehashes but doesn't cite essential work of 1999-2015". Which I guess does mean people with real "ideas" are also fed up with him?
"Deep learning pioneer Jürgen Schmidhuber, author of the commercially ubiquitous LSTM neural network, arguably has even more right to be pissed [...]"
He's (at least to some extent) arguing that if you're going to say someone's paper is "mostly wrong", saying the same things 4 years later should probably warrant a "ok, you were right" at least.
He's proven time and time again that he doesn't understand the methods at work and doesn't even seem interested in trying to do so.
Why? If their arguments are sound, why shouldn't we listen to them?
Where is this silly credentialism coming from?
>Why? If their arguments are sound, why shouldn't we listen to them?
Generally arguments from non-experts like this fall into the "not even wrong" category and don't merit much attention.
How exactly holding the SOTA record of any, or even all, machine learning task gives you any authority on true intelligence?
What gives LeCun any authority on true intelligence?
Even Lecun points out that his paper is not technical.
The only thing that matters is the of the argument.
I swear same thing was being said 10+ years ago
So I guess I'd say that if Gary has a legitimate beef, it would just be in regards to acknowledgement / citation / whatever. If Yann really was familiar with Gary's older work, then came around to the same ideas, but refused to acknowledge Gary, that could be seen as somewhat petty and vindictive. That said, I have no idea to what extent that is actually the case. Not trying to take sides here. I respect both guys to a tremendous degree.
Consider this:
LeCun, 2022: Today's AI approaches will never lead to true intelligence (reported in the headline, not a verbatim quote); Marcus, 2018: “deep learning must be supplemented by other techniques if we are to reach artificial general intelligence.”
How can that be something that LeCun did not give Marcus credit for? It is borderline self evident, and people have been saying similar things since neural networks were invented. This would only be news if LeCun had said that "neural nets are all you need" (literally, not as a reference to the title of the transformers paper).
And furthermore, if LeCun had said that, there are literally dozens of people who have also said that you need to combine the approaches.
He cites a single line:'LeCun spent part of his career bashing symbols; his collaborator Geoff Hinton even more so, Their jointly written 2015 review of deep learning ends by saying that they “new paradigms are needed to replace rule-based manipulation of symbolic expressions.”'
Well, sure because symbol processing alone is not the answer either. We need to replace it with some hybrid. How is this a contradiction?
To summarize: people have been looking for a productive way to combine symbolic and statistical systems -- there are in fact many such systems proposed with varying degrees of success. LeCun agrees with this approach (no one has anything to lose by endorsing adding things to any model), but Marcus insists he came up with it and he should be cited.
Ugh.
In his mind he is always right. Every single tweet he made, every single sentence he has said is never wrong. He is 100% right everyone else is 100% wrong.
In other words, Gary Marcus has managed to match some linguistic sub-patterns between two articles, but has not proved he is intelligent.
If you think that is substantive evidence for a stolen idea then it's surely not possible for anyone to ever have an original thought.
He says in the article that he doesn't think it's a stolen idea:
"I won’t accuse LeCun of plagiarism, because I think he probably reached these conclusions honestly, after recognizing the failures of current architectures."
Why do people keep upvoting his stuff?