Shush, this is Hackernews. We all want to have our egos stoked that our industry is the most important in human history and that tech will transform and save us all. Go away with your historical analysis /s
281 karma · joined March 29, 2023
There were nuclear researchers who did not pay into their retirement fund since they thought it was kinda high chance humanity extinction was happening in their lifetime during the heyday of nuclear proliferation. We are seeing the same play out in AI. I know at least one AI researcher who is not paying into their retirement fund.
Never ascribe to malice that can easily be explained by game theory tragedy of the commons or prisoner dilemma type dynamics. Race dynamics and systems theory is one helluva drug and leads to horrific outcomes all the time without the need to ascribe malice to any individual actor.
The US military asks me to build the super bioweapon. I say no thanks, please ask someone else, but happy to keep selling you bioweapons I think is safe enough.
I get designated an evil unamerican supply chain risk that nobody in the US military may do business with even though I was still happy to sell my safer bioweapons.
That is the tl;dr
The US military asks me to build the super bioweapon. I say no thanks, please ask someone else.
Am I a supply chain risk?
It's the difference between a sword and a nuclear missile even though both are "technology"
Imagine if the biotech industry had most leaders tell everyone publicly that what they are building has a high chance of killing everyone and that there are huge risks. If there were people online saying the biotech industry is just fear mongering for investor signalling and regulatory capture you'd role your eyes at the online commentators for their Dunning Kruger effect lack of understanding on how dangerous man-made biological agents can be.
What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.
""" One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:
> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”
You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.
These people don't give a shit and aren't taking things seriously at all.
Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.
One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.
The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1. """
And then since they are an intellectual nerdy bunch who highly value their own IQ any time AI does something unexpected they didn't predict would happen so soon, said engineers fall back on "they aren't really conscious tho or it isn't really intelligence unlike what I have in my human brain and that distinction matters".
And then go on to completely ignore the thing that is actually important: AI capabilities.
AIs could escape an engineer's containment en masse as a swarm to some other server, psyop an engineer into giving it money, hire a hitman on the dark web to murder that engineer's child and he will still say there is no serious risk to all of humanity.
Signed by many AI researchers with no financial stake in the industry. It comes across as hyperbole because you're not an insider and haven't see what they have seen.
Global warming would as well if you were only learning about it now as a newer thing only some scientists were concerned about. But unlike AI risk, global warming has had decades to settle into the cultural overton window.
Either way, the biorisk has orders of magnitude more evidence than wild basilisk speculation. Please do not put those in the same category.
Also your evidence-first approach and no concern without evidence falls you into the turkey problem. Imagine being a turkey and being fed each day. Some other x-risk turkey is saying that the situation is suspicious, that the red barn is where some well-fed turkeys have been taken and none return and that this is happening less and food has increased so maybe all of us will go to barn at once soon. You tell the x-risk turkey they have no evidence a lot of turkeys have ever been sent to the barn which shuts them up. Reasoning by induction would get you "we are happy and thriving 99 days so should be good on the 100th" and then you get slaughtered on the 100th.
Consider the importance of first principles approach where you deduce the possibility of events that will happen only once and never again (like human extinction). Nobody with a "show me the historical evidence" approach would have been able to see the industrial revolution coming.
I think the gradual disempowerment thesis here is well argued enough to be considered the default track for what will happen to us: https://gradual-disempowerment.ai/
If you disagree with it, where do you disagree with it?
Biotech - "what we are building our noble prize winning expertd say will likely will end humanity, wanna buy shares?" Oil - "this will likely lead to the end of civilization, 20% of leaders in the field say so, wanna buy shares?"
I keep seeing this take that this is a marketing stunt. The burden of proof is on those that say so. The most parsimonious explanation is simply that real experts in AI believe the risk is very real, and not for ideological reasons.
So much so that I worry they won't be Machiavellian enough to survive. Hope I am wrong.
Same goes for anyone saying all gambling is the same.
This is because bacon is more like cigarettes than most people assume, even if far less dangerous in practice.
Like another example is "sugar is poison." which is also structured as a factual equivalence and also gesturing at something real and also designed to land as a stronger claim than the evidence warrants.
If you want to simply go by societal resilience from biorisks then switching to more easily controllable substances like plant based meat for protein would be an absolute win.
The Trump admin acts like it is on cocaine. Many people - and I think this can be a highly rational preference - prefer predictable more evil of chaotic less evil.