Please ignore the deluge of complete nonsense about Q*
twitter.com
twitter.com
>"The board didn't handle things well, but they were right to be concerned because OpenAI did have some sort of research breakthrough"
Not coincidence that this leaks after Sam comes back, rather than before when it could have made the board look more justified in their decision. This changes the story from incompetence to "it's a problem only OpenAI has because they are so far ahead and close to AGI". Masterful PR move to leak this and shift the narrative
That's not to say it must be deeper than the usual power plays and politics, but breaking an NDA doesn't seem to fit well fromwhat I've seen.
Interesting claim. We know with certainty that there was a conspiracy to move against Sam Altman. What does "conspiracy theory" mean?
In actual "language game" usage, it's just a derogatory term meaning "I don't like these unproven claims, even though I can't disprove them".
Such as casting clear evidence toward the “safety/commercialization of next gen tech was a point of contention among the board” thesis instead as:
> The leak about Q* feels like an olive branch to let the former OpenAI board and Ilya save a bit of face, probably part of the terms for Sam coming back plus it distracts from all the drama and puts a positive spin on things.
This seems like it would apply equally well to all theories.
And the conclusion you draw is fully general. There are more possibilities for why something happened than there are reasons for it. What are we supposed to have learned?
> Especially if, as ethanbond indicates, the person with the theory doesn't believe what the other people say about their motives.
The modal case is that, when someone makes a claim about their own motives, they are lying.
The second most common case after that is that they don't know their own motives and their reported motive is not accurate.
> conspiracy consists of one or more people cooperating with "each other" in public.
Public doesn't have to have anything to do with it.
> What are we supposed to have learned?
Yes, I overgeneralized. In general the thing to learn is to try to reality check as much as possible.
> The modal case is that, when someone makes a claim about their own motives, they are lying.
Granting this point, a razor would attribute base motivations over complex motivations.
Does anyone really question what the parent commenter was referring to with the phrase "conspiracy theory" given the context of this thread?
In contrast to that have been these wild theories that are mutually exclusive claiming to know the beliefs, emotions, and plans of practically anybody involved.
In all I have found it to reveal far more about the speculators than it has about events themselves.
p.s., maybe bomb us a good idea.
It's to save face.
It's all a cover to hide a big R&D discovery they made but don't want to reveal, so they made it about personal dispute with Sam.
It's a marketing ploy to hype people up about a new OpenAI product.
etc etc
Hell, there's wars killing tens of thousands of people going on right now, and a ton of money is changing hands making a juicy business for whole industries.
How easily smart people convince themselves of what they want to be true with zero self-awareness makes me much more fearful of what's to come.
>other side gives, imho, convoluted arguments and asks for them to be proven wrong (as opposed to trying to prove themselves right).
The question is what should our default stance be until proven otherwise? I submit it is not to continue building the potentially world-ending technology.
When you see a gun on the table, what do you do? You assume its loaded until proven otherwise. For some reason, those who imagine AI will usher in some tech-utopia not only assume the gun is empty, but that pulling the trigger will bring forth endless prosperity. It's rather insane actually.
That’s not a rational argument for why we should be concerned — you merely asserted you were.
Why is catering to your feelings the default position?
The fact is we don't know how current ML actually does what it does, we don't know what we'll have next month, and we wouldn't know how to recognize an AI or AGI if we developed one. The risks and unknowns are high, the default position should be to not develop the technology unless and until someone proves without reasonable doubt why we can and should do it, and how we'll do it safely.
Right, and Yann is arguing the point that AI and LLMs are not or will not be dangerous. Where's his proof? As the parent posters have said, he has none.
LLMs are tools in amplifying individual human intelligence, 100% of automony and will come from their user (human).
Also, even as tools they have fundamental limitations which stem from their autoregressive nature
Your argument, and Yann's, is that AGI, or what you call AGI, is a kind of quasi-intelligent golem that, despite being generally intelligent, doesn't have human-level intelligence. Your claim that it will never be human-equivalent, much less trans-human/ASI, is built into your worldview. It's not a conclusion. It's an assumption on your part.
People like you and Yann can believe that if you want, but you have no evidence, because nobody knows what's required for human-level intelligence. Nobody knows whether some kind of system involving neural nets could develop human-level intelligence or beyond. It could involve different architecture or training methods. There's no assumption by AI doomers that AGI will be achieved by a LLM with more parameters or more or better training data.
The only approach we will be able to claim objectively can produce systems with human-level intelligence is procreation.
Of course over 100 years later we know it's not only possible but manned flight is far more capable than natural flight in almost every metric.
https://promptengineering.org/what-are-large-language-model-...
I think this whole quote is riddled with assumptions in this debate.
What is human level? Is it really a "level" or is human level just a local variant in a space of possible intelligence varieties that maybe could be sorted along one dimension of levels of maybe multiple? Can it be super intelligent without being autonomous at all?
I'm not saying that you're wrong in just pointing out where people ought to have a myriad of different assumptions.
So, more Ringo than Lenon?
An early GPT-4 test ended up with GPT successfully solving capchas by tricking a TaskRabbit worker into doing it for them [1]. When asked by the worker if it was a robot, GPT decided to lie to the worker and claim it had a visual impairment that made it difficult to solve the puzzle. That sounds like a level of autonomy and social engineering skills that could be concerning to a reasonable person.
[1] https://www.businessinsider.com/gpt4-openai-chatgpt-taskrabb...
If it asks for the right to vote?
Here you are, trying to "outthink" an AI and speaking as if you understand (both the AI and the world of politics at the very least)! Isn't that silly!
Autonomy and self-sufficiency are not the only ways a system can be dangerous.
But even if this claim were true, ChaosGPT proves that some humans will almost immediately set about using such a non-autonomous tool to create a dangerous autonomous agent. This is my problem with LeCunn, nearly all of his points are trivially refuted by real world observations, yet he keeps repeating them as if they simply must be true.
> Also, even as tools they have fundamental limitations which stem from their autoregressive nature
That's yet another speculative point that LeCunn constantly asserts. Scaling laws have not shown any indication of even approaching a limit.
People dismiss LLMs because they are not embodied, and lack continuous training. That is to come.
I only see his posts on Twitter but haven't been impressed.
It is the same with AI/AGI. Anything closed source and having regulatory oversight is useless, decreases innovation, increases bureaucracy and will only serve those who wish to build a “moat” to further their hold on the technology.
Source isn't even the problem, unless you're a billionaire, you can't afford to train the model.
This isn't the wheel, the printing press, the PC, or the public internet, which created opportunities for everyone.
This is pay-to-play that is only affordable to the nation-state the mega-corporation, and the latter might let you play around as a digital sharecropper on their platform, until they cut you off, because they can make more money by having first-party ownership of whatever you built.
Open source would mean someone could see and run the model code locally/independently to create an instance. For an LLM, this is much less insurmountable of an issue in terms of resources needed.
For example, I run a few instances of open source software though my instances' databases are private.
We are still answering questions on LLM/AI scaling. Ya, you might have your cottage industry AI giving out answers on the trickle of information you feed it, but will that even be comparable to one being fed megawatts of power with terabytes of data per second flowing into its databanks?
There are many sources like him, often with even more detail and references. The problem is that these people are not famous and thus do not have as many followers and are less likely to make it into your feeds. Lots of grad students are in this region as they write to help increase their name and visibility due to the need to market one's self. Don't be afraid to look at those who are not from big name schools. Instead look for those that are willing to mention details and nuance. But this may be hard to accurately determine being on the outside, but that's the thing hype people take advantage of. I don't think they're malicious, but if you're the "smartest person in the room" it's easy to think you're the smartest person in many rooms and have a high confidence in bullshit. For example, we see such comments on places like Reddit and HN. I'm not sure if there's a good way to realistically filter out non-experts from experts (I don't think we should require credential checking). But experts are probably usually more boring as is reality to fantasy.
Your verbiage isn't in the spirit of HN.
NB: I know nothing about these people and don't track it, but my consistency check failed on this claim based on the flood of news over the past week.
Patrick Collison seemed satisfied with Sam's tech chops [1]. If that isn't enough of a vouch, god help the rest of us.
[1] https://www.quora.com/Is-Sam-Altman-highly-technical-Has-he-...
Also, I wouldn't call it a myth. I would call it 'questions around his technical knowledge'.
The best way to answer those questions would be for direct evidence of his technical ability to be shown, not references from mates (just as any interview for a tech position requires direct evidence).
"Elon Musk doesn't know how to code" is my fav. It's as if coding is some sort of mythical and rare skill. A new grad who just learn to code a year ago is able to intern at FAANG, but Elon Musk cannot possibly learn how to code for some reason.
My guess is that they arrived to an inflection point. The company was on the cusp of taking an investment which would have valued it at 80 billion. One of the board members mentioned that self destruction would have been in line with their mission of safety for humanity. It seems like at some point you have to ditch the "we're doing this for the benefit of humanity" act and act like a profit hungry machine with a startup CEO going balls out. In this case, "balls out" won. The board was way too late. The AGI escaped with the creation of the for-profit company.
Would have hardly been a shock that "after an exhaustive search for the next CEO, we have decided the most capable person to replace the interim CEO is low and behold Ilya Sutskever."
The fact even I know Sutskever signed the petition is just another aspect of this Shakespearean drama, Sutskever doth protest too much me thinks.
// nitpick, it's lo (from look) not low
What they've accomplished so far is really impressive. But until I see AGI for real I won't hold my breath.
Just feels to me like some expert/wizard level PR is going on right now.
To the average person chatGPT is literally magic.
Again, there's plenty of different conclusions that can be made on the risk levels but that doesn't change his expectation on timing of development.
He is basically the next generation's Ray Kurzweil but instead of AI granting us biological immortality now AI is going to cause mass extinction.
Instead of uploading consciousness to some digital nirvana we get the slightest variation on The Matrix. "And then it takes over the galaxy and uses the atoms in our bodies for batteries!!"
With the way AI captures the imagination we can't but help take things to irrational extremes.
From everything I've seen from him, his concerns revolve around (1) a lack of a bounding function for the development of AI (2) a view that the alignment problem is unsolvable and (3) the risk that we won't recognize an AI was developed until it is sufficiently more intelligent than us to pose a fundamental risk.
I raised it only as an example of a prediction he made that most at the time thought had no chance. My point isn't to claim he's some genius that gets everything right, but there are examples of him making predictions widely viewed at the time of having no chance that turned out to be correct.
Writing him off because people responding strongly to his predictions of AI destroying us all doesn't do anyone any good. His core arguments and predictions seem sound and I haven't yet found anyone to explain how we can solve alignment after the fact, for example
I'm writing him off because he isn't a practitioner. I can make things up about what will or won't happen on my own, as can anyone else.
If I want to believe every wacky concept from science fiction will happen, I don't need a middleman to tell me it will.
It does feel risky to write off anyone that isn't a practitioner. There won't be any practitioners working on AI risk, its a theoretical pursuit by nature. The closest a practitioner could get is AI risk management, i.e. attempting to limit risk for AI we're already developing.
Taken in that light, his being correct about his protein folding prediction feels like cold comfort when taken in context of his predictions, largely unsubstantiated, of AGI becoming a malevolent human extinction force at par with or of greater risk than that atomic bomb. It elides the very real conceptual limitations of current implementations of AI, such as it's inability to recognizing bidirectional associations (Tom is John's son means John is Tom's father), protecting against injection risks, "hallucinations" (inability to differentiate between when it is making something up and when it is not) and a variety of other existential limits towards its own power.
These details, far from being inconvenient nuances that do not detract from his overall point, should actually serve to make us seriously skeptical of how well he understands the risks of the entity he is warning against. A more skeptical observer might even say (and I'm one of them) that much like fire and brimstone preachers, he is trying to create an unrealistic fear of a deific entity that cannot be reasoned against so that he can consolidate power by serving a "solution" that protects against it and curry popular favor. I find his approach to be the distasteful and manipulative approach of the common demagogue. The haughty, arrogant, insecure way that he treats the arguments of his critics (he simply blocks them on Twitter or condescends to them) make me even more skeptical of him. It makes me wonder if on some level he knows that his argument is not being made in good faith, and isn't trying to win an argument in good faith.
How many of his predictions are even disprovable? Do you have a list? Was that protien fold prediction even disprovable?
An example of a nondisprovable prediction is:
-one day an AI will become self aware.
A disprovable one is:
-Skynet will begin to learn at a geometric rate. It will becomes self-aware at 2:14 a.m. Eastern time, August 29th 1997.
When we have AGI, we will find it out pretty quickly since whoever gets there first wants the most money and control they can.
If someone can build it someone will. Laws and impending Doom or not. It's probably going to be better if it's us than Russia or China.
What they can do is ensure that the sanctioned entities are always at least a generation behind. That's not especially helpful in the context of "anti-doomers will be wrong exactly once" unfortunately.
LOL and how is that working?
No reason to believe this, it hasn't been true for any other dangerous technologies like biological weapons or nuclear power.
LeCun: “[Note: I've been advocating for deep learning architecture capable of planning since 2016].”
Reply: “My understanding is Schmidhuber already solved that 10 years ago. Just no-one knows it yet.”
From 2015. https://arxiv.org/abs/1511.05440
Edit: I didn't realize you were sarcastic (－‸ლ)
Isn't it more plausible that someone is just piecing together a couple key public papers in a novel way?
But doing it on the scale of something like GPT-4 is only something OAI can do, and I don't doubt that there's some breakthrough there. But IMO it's likely that the breakthrough isn't algorithmic, but a result of scale, like most of the other OAI breakthroughs.
It's shocking to me that people think Google and Meta don't have the scale to operate at this level.
The only way you could catch up faster is by having orders of magnitude more hardware. But due the global shortage on hardware right now, nobody has that, not even Google.
This reads like wishful thinking, in the absence of a real moat. Do you think the hyperscalers who are buiding data centers as quickly as they can get permits do not have the infra scale to match OpenAI? Who do you suppose has been providing ever-increasing revenue to Nvidia these past few years? They didn't start doing ML experimentation when ChatGPT was announced.
I agree that they aren't specifically giving LLMs all of their attention - their infrastructure still has to support existing, profitable products that bring in real revenue. I disagree that they can't experiment to the same scale.
Google has nearly unlimited resources to pursue this shit.
One project was to implement a simple Q-learning action/value system to play simple games, like Pacman.
The crypto-bros-turned-AGI-experts on twitter are spouting the most uninformed, misguided garbage about this whole thing, it's quite amazing to watch.
And I'm not saying that I am smart or an expert about Q* because I took an introductory college course. I'm saying that even I, someone who knows basically nothing beyond the introductory concept can identity that these people have no clue what they are talking about, and yet the have this incredible talent of speaking in such an authoritative and faux-intelligent tone. It's amazing.
My favorites are the tweets that sound like this:
"So, now we know that [insert something totally wrong]. Well, what if extend that further, by [another totally wrong conclusion]. Here's an explanation of how this all works. A thread, 1/N"
followed by a full thread, images included, of drivel.
I miss the good old days with those idiots were mostly stuck to the honeypot that is/was cryptocurrencies.
After the LK-99 debacle where Twitter "confirmed" its superconductivity and breathless sheisters heralded the dawn of a new age, my new policy is "inverse Twitter": if Twitter thinks something is a big deal, then it's more than than 50% likely that it's unsubstantiated horsepucky. The signal:noise ratio on Twitter has always been uselessly low, but the post-crypto scene has plumbed heretofore unfathomable depths.
No clue why people try to come up with these obviously silly heuristics.
Making a decision for oneself is totally reasonable, but there should be a high bar as soon as someone tries to push that as a conclusion that others should follow.
The problem during the pandemic response wasn't that individuals tried to understand and decide for themselves, its that those who were supposed experts got out over their skis and were forcing decisions and conclusions that weren't backed up by data. What's wrong with someone reading studies and deciding whether they themselves want to get vaccinated or west masks, especially when we didn't have solid data to show how those decisions might impact others' rights?
I could not imagine the same amount of searches and guesswork had they called it "P*" instead.
It seems like Q inherently has some mystery to it. Maybe in that is the least commonly used letter and it rarely is used separately from "Qu"
I ended up unfollowing him as the curmudgeony approach combined with some of the far out stuff was just a bit too much for me. Doesn't really improve discourse in the field.
Yann is a nobody at Meta (lol at calling FAIR a top lab at this point, it’s not even the top org in Meta for AI) and keeps trying to claim credit by association for Meta LLM success.
The lecture in which he mentioned it is this one:
https://memento.epfl.ch/event/campus-lecture-yann-lecun/
Comparing with Schmidthuber is a bit strange too, he claims everybody is doing it.
Ane even more conspiracy-minded take - from the Russian news naturally - is that it is the Great Battle for the future of humanity between "doomers" (Oh, no! the AI is going to kill us all, we need to stop all the work and control the GPUs like guns) and "Effective Altruists" (We can do all the evil today in order to achieve greater good tomorrow)
Close. It’s to generate hype for the upcoming $86 billion share sale.
Anyone have a good source for learning more about this topic?
https://github.com/AGI-Edgerunners/LLM-Planning-Papers
Here's one titled "Reasoning with Language Model is Planning with World Model"
https://arxiv.org/pdf/2305.20050.pdf
It’s different from the link the other user posted.
This one is specifically about the Q* process
He’s been consistently voicing realistic statements with regards to AGI (he doesn’t believe it’s a thing), AI safety and doomerism, limitations of LLMs, etc.
Lost respect for his opinions massively after seeing how he lied about the topic.
And that’s true now as ever. I also heard him say that training multimodal models on text+image/video would mitigate the grounding issue, and that’s proven to be true too.
So I’m not sure exactly what your objection is.
His research at meta is in analytical methods of AI so I'm not surprised. LLMs were seen as anathema to him as he could not comprehend the capabilities that would come from simply scaling up a suitable architecture with compute and data.
It's quite remarkable how confidently incorrect he was proven to be. Now he's going for round two on his takes with AI safety. Clearly he didn't learn his lesson.
https://thealgorithmicbridge.substack.com/p/gpt-4-a-viral-ca...
Q star would mean optimal Q. If that is even what the star is referring to. There is still no way to know if the Q is the quality factor, query value, question, something else, or just a placeholder name that doesn't reference its structure at all
There may be a breakthrough, there may not, but nothing on the topic is convincing or worth reading.
> Reuters could not independently verify the capabilities of Q* claimed by the researchers.
True, they are a reputable news agency, but the parent is also correct, it's not highly credible.
I agree that the reporting on Q* has mostly been nonsense, and I suspect those articles were published just for the clicks, but at the same time this tweet makes zero sense because an AI model which can plan is somewhat concerning so perhaps those articles were on to something?