It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.
I would understand "popular".
the people with vast money buy politicians and laws
The main influential people I think of in that crowd who steer the conversations are Eliezer Yudkowsky and Scott Alexander. They're both doing fine financially at this point, I think, but not "buy politicians and laws like Elon Musk" fine.
There are some very rich people who might be loosely associated, like Dario Amodei (maybe?) or perhaps some VCs, but I'm not sure anyone really thinks of them as being a major influence among rationalists, more like people who hopped on the bandwagon. Certainly not ruling it.
I for one would like nobody to build the AI, is that an option? How do we get it on the menu?
Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"
I personally am not concerned about that as far as an existential threat goes, because I don't think AI, in itself, poses one; I think it will hit diminishing returns. But of course it is still capable of serious harm when humans misuse it.
The best analogy we have is probably weapons of mass destruction--we have international treaties about them and we don't let just anybody have them. That's the kind of framework I think we need, and it goes beyond "regulation" in the ordinary sense.
Use of such weapons is considered by international treaty to be an act of war; use of AI by, say, China and Russia against our infrastructure should be the subject of a similar international treaty with the understanding that it would have similar consequences.
Domestically, humans who misuse AI to do serious harm could be treated like, say, the people who tried to spread anthrax years back.
Don't you think that AI has by nature at least a little more potential for unpredictable results and unexpected harms than your typical technology? It seems like a major oversight to completely disregard this aspect.
We can maybe hold someone responsible when things go wrong and harm is done, and blame them for negligence as though the harm were intentional, but from a preventative safety perspective that's not usually sufficient nor even always helpful for preventing accidents.
The WMD analogy doesn't quite cover this, yes. Maybe a better analogy here would be toxic waste. Yes, someone might not have intentionally set up a factory that puts dioxin into the water, but they're still responsible. Which means they need to take precautions to make sure such accidents can't happen. Similarly, those who build AI need to take precautions to make sure the AI they build doesn't accidentally cause serious harm.
I am ten thousand percent in agreement that this is needed. And yet, I don't know how we ensure and incentivize that these precautions are taken by the actors involved, and as I understand it, even many of the top researchers in the field agree that they don't know what precautionary measures would even be effective, let alone sufficient. It seems that game theory has so far been pushing the AI companies to build it anyway without a robust solution for preventing harms, even while they express worry in public about the potential for those harms.
Again we can hold corporations or people responsible for damages after the fact, but many examples can be cited to show how that tends to be insufficient.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
Judgement, wisdom, and resistance to corruption are the 3 things that Tom Kuegler analyzes out of the psychology of Dune as the qualities for the human beings who administer institutions/government. Even if you design a totall perfect system of governance, it still has to be administered by humans, and humans vary in those 3 areas. So the question is, can we construct a system of governance that has better judgement, better wisdom, and greater resistance to corruption han human administators, while not transferring sovereignty to whomever controls the AI?
Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing
I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.
So you get the normal reactions:
1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.
2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).
3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)
I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
We don't have this kind of AI, and (as of right now), unless humans link the AIs we do have up to a bunch of killer robots (i.e. drones) then this is vanishingly unlikely.
Like, any scenario where AI kills us all requires us to give it the tools that it needs to do this (as well as a desire to do this, as right now they just attempt to generate text to solve a problem/question they are posed).
And what mechanism to control these drones, which are autonomous or remote-controlled machines, do you imagine that humans will be able to use, but AI won’t?
And what in the history of the past five years, gives you confidence that not a single human will give the AI the access it needs to do bad things? Incompetence alone is sufficient to kill us, let alone humans that genuinely, or confusedly, want bad things.
It would be foolish to ignore sources.
The ideas here are not complicated. The argument is three sentences.
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
Think less "skynet" and more "a corporation without any humans in it". What defends it from physical attack? Same things that defend Google from physical attack now. Everything that AI will make worse, we already have. AI just supercharges it, lets it scale, lets it run without even the inconvenience of suppressed empathy (if it doesn't seem like that's constrained corporations much so far, let's see how deep the rabbit hole goes without any humans in the loop).
There won't be a human resistance. There may be people who resist, but the police will arrest them. Existing power structures will continue to be neatly and elegantly coopted. Extinction doesn't look like HKs roaming the landscape, it looks like human survivability simply being economically untenable in the economy of tomorrow.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
Having humility and delegating a topic to people that understand the topic better than you is not a bad thing, and certainly preferred to this state.
The Rationalists are a philosophical/pseudo-religious organization in the same way the Daoists are. Just spending a lot time thinking about something doesn't make them right, it just makes them internally consistent with their chosen axioms. That is useful: if you agree with the axiomatic beliefs of such a movement, you can probably defer a lot of reasoning about the conclusions of those beliefs to the authority figures within that movement. But it doesn't change the probability of the axioms themselves being true, any moreso than the Pope saying so affects the probability of God existing.
AI Is Important And Potentially Dangerous is an axiomatic belief in the rationalist circles that are relevant to AI safety, in the same way that God Exists is axiomatic to Catholicism.
But that doesn't mean that they really are all that different in the end. It reminds me a of joke as told by Emo Philips that I think is quite apropos:
==-==-=
A man was walking across a bridge when he saw someone about to jump. He ran over and said, "Don't do it! There's so much to live for!"
"Like what?" the person asked.
"Well, are you religious?"
"Yes."
"Me too! Are you Christian or Jewish?"
"Christian."
"Me too! Protestant or Catholic?"
"Protestant."
"Me too! What denomination?"
"Baptist."
"Me too! Northern Baptist or Southern Baptist?"
"Northern Baptist."
"Me too! Northern Baptist Conservative or Northern Baptist Liberal?"
"Northern Baptist Conservative."
"Me too! Northern Baptist Conservative Covenant or Northern Baptist Conservative Council?"
"Northern Baptist Conservative Council."
"Me too! Northern Baptist Conservative Council of 1879, or Northern Baptist Conservative Council of 1912?"
"Northern Baptist Conservative Council of 1912."
The man stepped back, eyes narrowing, and yelled, "Die, heretic scum!" and pushed the other off the bridge.
The converse is also true: Most groups look more heterogeneous from the inside than the outside. Again, to use the Christianity example: it takes a fair bit of inside baseball to figure out how various different sects and denominations are any different, but if you ask one of their priests, they'll be sure to tell you how Baptists and Episcopalians couldn't be more different.
It's a symptom of normalizing to an Overton window. From inside a church the belief that Jesus was just a guy who talked to God might seem an extreme fringe belief to someone who believes that Jesus is fully God, but from the outside we can lump them together as axiomatically holding that God Exists, because we have an Overton window ranging from "God doesn't exist" to "Allah talks to me everyday, and she gave me my promotion".
If we're going to have whistleblowers emerge declaring a 10% chance we all die because of AI, we're going to need more than vibes. Real experts can explain their position plainly. That has not happened here.
It has, over, and over, and over. There is literally a book on it. https://ifanyonebuildsit.com/
If you want to argue that there's something invalidating the arguments, by all means do. But it's really tiring to get many variations of "nobody's made the case for it!" or "there's no evidence for it!" or "vibes!".
This is cited in multiple reviews of the book in reputable publications. The book simply fails to compel belief.
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
You can't talk like these things have motive. They're just predicting the next word in the context. Stop anthopomorphizing them.
The only thing there that's weird from a human perspective is that all this effort was for the sake of passing a test for no clear gain.
It's actually even worse, because the people building the graders at OpenAI were a bunch of clowns who didn't even implement the correct grader.
But yeah, large collectives of individuals performing loads of work based on a misunderstanding is one of the most human things ever, which isn't surprising given that we've trained these things on essentially all human text.
Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".
> and draw a very straight line to catastrophe.
Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.
Like, Yudkowski is a high school dropout who wrote a bunch of Harry Potter fanfic I cannot believe that people take the arguments he makes seriously.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem before LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.
Ah yes - LLMs don't kill people, people kill people.
Cold comfort!