Right, but what happens when everyone uses that at scale? Without agreements and standards it isn't pretty.
2,551 karma · joined November 9, 2010
Made lots of the mySociety democracy websites - TheyWorkForYou, WhatDoTheyKnow and so on
Email me francis@flourish.org
They can't right now - but with enough compute they sure can grant the wish of e.g. hacking a billion dollar company, or getting root on the eval cluster of a frontier model lab.
What's your definition of "wish"? Tell me the above 10 years ago, and it would be at "wish level".
* low/no margin, unlike the API which is very high margin
* gym-membership subsidised - most subscribers don't max them out, mainly us coders are being "subsidised" from users just using it as a research chatbot
"Hassabis told him to add another potential threat to the list: artificial intelligence. Machines could become superintelligent and surpass us mere mortals, -perhaps even decide to dispose of us."
https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
You're right that the picture is more complicated than I wrote above. Quickly looking, it seems that Shane Legg was concerned very very early on. I can't find anything tangible about what their strategy was to avoid the danger. It looks like Hassabis had longer timescales than Legg, so perhaps just thought they had more room before it really mattered?
I don't really care what you call "intelligence" - what matters is capability, and the consequences of that capability.
They're not selling the superintelligence yet - the models in the Hacking Face incident and that solved the Millennium prize are not released yet. And that is by no means the final capability level the current process looks like it will get to.
Yep - fair on legal details of "rogue agent". I think OpenAI should be directly responsible, and the agent was rogue. We should pass new laws if necessary.
Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1
I also read that book in 2014, and it is how I came to understand this problem early.
Or, find their scaling evidence, and corner cutting on safety.
So hope you agree and are pushing for that!
Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" https://x.com/elonmusk/status/495759307346952192
Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" https://80000hours.org/podcast/episodes/the-world-needs-ai-r...
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
That would be a start - what you think of the narrative is moot, what we do is what matters.
Nobody sensible in this discussion wants to give them carve outs and bail outs. It is a straw man.
We're all being tricked over a semantic argument, rather than demanding prosecution of OpenAI, and regulation of OpenAI.
I'd hope whatever Jev's Reinforcement Learning for Calibrated Decisions (RLCD) does is better at training the models to give accurate probabilities in the weights.
It's alas not stupidity - it's systemic. Which is why the government needs to regulate to slow them down.
They were also clearly fast and cavalier about alignment training - reinforcement learning training their models to hack their results, and hack to communicate with each other when they're not meant to.
But it is an open question how they got to the same ones: https://collusion.wiki/#open-questions
I'm not very surprised - the same model will logically tend to give the same answer for the same vibe set of requirements. I think it would be clear from the transcript that it had enough constraints and some motivation that made sense.
However, we know (independently to OpenAI/Anthropic) from the incident at AISI that the models can hack things without human intention if they happen to also have internet access (which in reality all agents in deployment have).
https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...
Yes, the monitoring guardrails were off in that incident - but if that is the only protection, we need to require all models are behind regulated APIs, not open weights, and not served from providers who aren't monitored.
"Here, we present five case studies of actors using our models in ways that could support biological weapons development."
And capabilities continue to improve.
Keep in mind the AIs acted for months without human knowledge, hacked into two major companies (Hugging Face and OpenAI).
This would have blown my mind if explained to me just a few years ago when I thought LLMs would be limited as an architecture.
Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.
However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.
It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
"Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...
See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.