OpenAI Dissolves High-Profile Safety Team After Chief Scientist Sutskever's Exit
bloomberg.com
bloomberg.com
Personally, I hope AGI is an unrealizable nerd fantasy, and OpenAI gets destroyed by Musk's lawsuit.
Cause thats whAt AI Girlfriend THEYre planninh
He was on the Dwarkesh podcast the other day and Dwarkesh asked him a lot of safety questions. He had very short timelines for superintelligence, and no real answers for any of the safety questions Dwarkesh asked.
Of course, there are still true believers in these companies. They were true believers years and decades ago. But everyone who was suddenly introduced to the concept is no longer interested as the frictions of real world progress become apparent.
It's like taking a star NFL/NBA team and sending them to play a season of exhibition games in a foreign market instead of continuing to compete in the peak league of their sport.
Those will happen whether or not you have a safety team. The best way to protect against these is to widely distribute deep fakes of famous people so everyone is aware that faking things is possible.
And there are already open source LLMs that can be trained to scam a significant number of people. OpenAI can't do anything about it.
The only remedy is social media warning people.
But that would have basically meant the company was wrapped up and the model was given away or sent to a working group to study in private or something.
Which would not be compatible with making money.
This is also why they did not demonstrate or even mention for one second the text-to-image capabilities of the latest model during the demo. Because it makes it too obvious that the model is truly general purpose and capable.
This is also why Altman made a big deal about it being free (to some degree) and explains the recent "feeling the AGI" photo on X/Twitter.
It also goes into the Morgan Chu lawsuit that is still going forward as far as I know. And you guys will see in the movie that comes out in a couple of years that my explanation was right.
Also, don't be surprised if they get Gal Gadot to play Murati.
I think super-alignment and super intelligence are incompatible with each other. I think a huge part of intelligence is in examining the basic premises you believe in and being able to work out the implications of that. Look at the Enlightenment and the Renaissance. I don’t think we would have had the Scientific Revolution without the intellectual underpinnings that reshaped how we saw the world.
In addition, super alignment is very hubristic. We are positing that our way of looking at the world is the correct and valid way and there is no better way that could be found even by a super-intelligent entity.
So either “super-alignment” creates entities that are so crippled that it would be hard to call them super intelligent, or else it is just the equivalent of putting an internet “safety” filter on the computer of a teenager that writes open source kernel drivers for fun - something that will quickly be bypassed.
I am glad to see this pseudo-religious BS finally being got rid of.
But we know that memory-augmented LLMs (not conventional LLMs, but a minor extension that will likely become commonplace, or some variation thereof) are Turing complete, so being able to inspect a model and guarantee some property is equivalent to the halting problem.
It's probably hard to justify a research team whose goal is solving the halting problem, which is provably undecidable in the general case.
It is important to disambiguate the scientific problem from the engineering problem. We can prove that lots of programs halt, not that all programs can halt.
It isn't a reason to not do the research. Sounds like an armchair way to not try.
Had they framed their research as "how can we design models that are limited enough that we can guarantee their safety" (rather than how can we design a powerful extrinsic inspector to supervise), that would have made total sense to me. But if you can do that, many of the motivations for superalignment don't exist. Put another way, implicit in the superalignment game is the idea that models that require superalignment are going to be at least as powerful as what's on the horizon, not some reduceable subset thereof.
I suppose the analogy for models is that if it can't decide whether output of another model is "safe", then it is not safe, so terminate. In practice, this could possibly be useful in the way a web server that times out after 1 second is still useful when its response times are normally measured in milliseconds.
(Assuming of course that safety means anything in the first place)
Models less power than a turing machine are still incredibly powerful. Some useful general purpose languages only accept programs that terminate. Hell, every useful program I ever wrote could be (in principle) proved to halt (excepting a `while True: do useful terminating thing again` main loop, of course).
Forget Climate Change, there's a reason the saying, "safety regulations are written in blood" exist. It's not that nobody ever forsees these issues, it's that people have a tendency to not care about future issues until they become present issues, no matter how sound the warning.
Nobody except these execs know if they're really disillusioned with the AGI Fantasy or if they just don't care about stalling business. As things stand, Open AI's charter incentivizes downplaying AGI for business gains.
1) you get an amazing demo that blows everybody's nips off after working in obscurity for many years.
2) everyone thinks that, because they just learned about the current state of the art it must have just been invented, and draws a mental line to AGI with a slope based on assumed rather than actual rate of progress
3) the grifters at the top capitalize on the hype to kite a whole bunch of cheques.
4) 18-24 months of everyone paying attention to every little micro advancement makes people realize that progress is actually as slow as it's ever been
5) welcome to the next AI winter.
I wonder if the level of investment was near comparable ever to what microsoft and google and fb etc are spending now on training foundation models.
I feel like the money guys really believe it this time, given the money we're seeing still being poured in, and it's not slowing down yet (please correct me if I'm wrong). but also I am biased as I love the idea of agi and desperately want this time to be different
Some more discussion: https://news.ycombinator.com/item?id=40390831
I think Microsoft might be Decima though, created by Evil Bill™.