Hacker News Reader

Comment by _yb2s | Hacker News Reader

Regardless of how realistic you think AGI risk is, it’s not marketing hype- it has been a human fear and central in scifi for a long time. Movies like 2001, Terminator, and the Matrix were hugely popular long before AI was profitable. Arguably ancient myths about golems and genies are essentially the same fear and concept. AI companies are afraid of public fear over AI as it might lead to regulating them out of existence…. They are actively creating marketing hype in the opposite direction, to convince people that AI is safe and useful.

View on HN

Fear of AGI is very real. But our proximity to AGI is marketing hype. In terms of existential crisis #2 and #3 are serious and already worth investing in mitigation. But #1 (AGI) is science fiction.

In 2022, the median ML researcher surveyed thought that there is a 5% or 10% chance of AI leading to "human extinction or similarly permanent and severe disempowerment of the human species," depending on how the question was asked.

https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...

I claim in 2024 this has doubled or tripled.

Fiction is much more than just entertainment... science fiction has been pivotal in both shaping and preparing our society for changes caused by technology. Much of the best science fiction has been so prescient that it's hard for modern audiences to even understand that these things were written before the world was like this. When I share a lot of old sci-fi with my young son, he finds it unremarkable, as they seem to simply be mundane stories about our present reality. Authors like Vernor Vinge wrote about AGI risk in stories because they were personally worried about it, and trying to share something they felt was important with others.

You can easily disagree with the warnings and fears shared in particular sci-fi, but to dismiss fiction as categorically irrelevant to our reality is just ignorant.

There is some important history here your comments suggest you are unaware of. One reason AI companies talk so much about AI risk despite it being bad for their bottom line isn't marketing hype, it's because many people in those companies are genuinely afraid. One could argue that this is because many of them have been exposed to the rationalist community- which could be (uncharitably?) seen as a doomsday cult obsessed with AGI risk. The founders of many AI startups including OpenAI were heavily influenced and active in this community.

I neither said nor implied that AGI being firmly in the realm of science fiction is a bad thing. There's nothing wrong with thinking about the implications and they would be significant. It's just such a stretch from where we currently are that it's not worth investing significant resources in mitigation. The threat from AI that is present and real is janky systems being depended upon for life and death decisions. Authors like Vernor Vinge (and I love his stories) depend on bending the rules of both physics and computation. Physics in that computation is somehow fundamentally different in different areas of the universe and computation in that known computability limits no longer apply.

It's easy to re-imagine science fiction as being closer to reality after the fact.

AI safety is an issue whether AGI is 10 or 100 years away.

Out of curiosity, could you recommend any Science Fiction written before the nuclear age that talks about nuclear proliferation? Or to take something from our present world, what Science Fiction stories would you recommend to talk about the effect of drone tech on trade and shipping?

Alas, Babylon was written in 1959. Not exactly before nuclear proliferation but before the Cuban Missile Crisis.

That wasn't what I was looking for, but I appreciate the honest response. UniverseHacker claims "science fiction has been pivotal in both shaping and preparing our society for changes caused by technology" and I want to understand that claim more. Let me explain:

By my reckoning, there are two driving factors in how the understanding and control of nuclear reactions affects our world: First, the huge step change in energetic density over chemical reaction, second the fact that you can cause explosive chain reactions. If there was a smooth gradient of progression towards the energetic density of Fat Man, the world would have been a very different place. If the science had panned out such that we couldn't cause a nuclear chain reaction, the bomb/energy source duality would never have emerged.

I'm curious if there is any fiction out there that contended with either of these factors prior to us realizing that the disparities between chemical and nuclear reactions are what they are. I've not read Alas, Babylon, and I'm not trying to deny that it had important influence, but I'm pretty sure we knew that it was possible to end the world via nuclear conflict prior to 1959. By your estimation, how did the book shape or prepare our society for the changes caused by nuclear technology? We're on the other side of the cold war and the conflict between capitalism and communism isn't a thing anymore, right? If we die in a nuclear fire, it will not be anything like, to quote wikipedia, "The explosion is mistaken for a large-scale US air assault on the military facility and, by the following day, the Soviet Union retaliates with its planned full-scale nuclear strike against the United States and its allies."

I'm not trying to be snarky, I'm trying to understand what I'm missing.

----

To me these claims of SF's prescience fall short, especially when it comes to the dangers of AGI. As an example, I have no mouth and I must scream, as wonderful as it is, doesn't have much to say about preparing our society for AGI. AM will not emerge from GPT-4o, and I'm pretty sure if AGI does happen it won't be through a super computer built to coordinate militaries. I enjoy reading SF, but I think its wider influence is as much miss as it is hit, and I think we can plot better courses when we treat SF as mostly entertainment. People who develop technology would be better served studying the history of technological development.

MAD and nuclear holocaust was a meme well before Kennedy. They were building bunkers in the mid-50s in back yards...

The unintended consequences of new powerful weapons falling into the wrong hands and new trade technology are very old themes in fiction even if the specific technical details are not.

I noticed you edited your initial response for tone before I could respond. Regardless, I'll respond to your initial statement to say that no, I'm not trying to make a snarky rebuttal. I'm trying to understand your statements better because my own personal experience and opinions disagree with what you claim and I'd like to integrate your perspective. I don't give a shit about internet points. You specifically mention Vernor Vinge, whom I haven't yet read. My understanding is his work is dependent upon their being an intelligence explosion with the development of AGI, which seems incredibly unlikely given how AI technology has developed thus far. I think we've seen amazing growth in capability but nothing close to an intelligence explosion as I've seen portrayed in SF. As someone working in this space, what lessons would I learn by reading Vinge's work as opposed to spending that time reading more about the history and happenings in the real world of computer security?

Now, to respond to your post-edit: I feel like my disconnect with your positions in this thread are around the importance of fiction. I think fiction is a minor muscle in the body that drives technology, if you'll allow the metaphor, whereas you claim "science fiction has been pivotal in both shaping and preparing our society for changes caused by technology." Perhaps you could explain what you mean more concretely? Another confusion I have: The claim of prescience and that the themes are very old seem to be at odds. If I develop an understanding of the dangers of runaway technology through a story about Genies, why do I need more modern work? What if I instead gain these understanding through actual history? The real world has so many more wrinkles than a story and so many more lessons to give about technological development than a story that I balk at words like pivotal. There is not pivot point, only hundreds and hundreds of articulations.

Anyways, I've written enough for now, and I don't expect you to follow up.

My apologies for the covert edit making you rewrite your reply.

Why is sci-fi useful when we could just look at technology history? It's the range of imagination involved- exploring big possibilities and unintended consequences that are unlike anything we've experienced before. A lot of it is also noticing how historical patterns could play out in a new context, and sharing those ideas.

I am reluctant to list out prescience examples, because on one hand they seem so numerous I couldn't do them justice with just a few, and on the other hand people will disagree with the importance of them.

Overall- I feel like it is a simple fact that lots of tech people and inventors love sci-fi, grow up reading it, and then it influences what they think about and choose to work on. I suspect we wouldn't have had rockets or space travel when we did, if we didn't have so many victorian era books imagining space travel. The numerous billionaire pet project space companies we have now seem a lot like people trying to live out fantasies from reading Heinlein. Even little things like the devices we have- my Kindle looks and acts exactly like the e-book readers in The Next Generation, for example.

When my son sees old Star Trek episodes very little is remarkable to him except the big things like warp drive and transporters. The "advanced tech" they use day to day is all stuff that is a normal part of his life- 3D printers, tablets, ebook readers, video conferencing, mobile communicators/phones, etc. Same with the "Back to the Future" movies- they travel to a future filled with various electronic gadgets that are very similar to those we now have, but didn't exist at the time... and my son doesn't see how remarkable that seemed to me as a kid, when those didn't exist.

As a researcher in biotech, some of my current research is definitely inspired by sci-fi, especially biotech and nanotech from Neal Stephenson books.

Edit: found this interesting list on wikipedia https://en.wikipedia.org/wiki/List_of_existing_technologies_...

Sorry for now responding for now, and sorry for responding so briefly, but I just wanted to say thank you for your earnest and thoughtful response. You rock! I don't have the freetime to properly articulate my rebuttal, but I will say we are on the same page in many regards. I think the power of story is very important in the process of technological development, but I think 1) All stories, not just sci-fi are important for developers and 2) stories are the least important and most pre-determined aspect of the process. Humans tell and learn from stories and act has purpose and use everywhere, but they are also easily misinterpetted and used as inspiration for badness. Most of the billionaire pet projects fit under this umbrella. IMO Great sci-fi inspiration should be the sprinkles on top of a layer cake of development prowess, experience, thoughtfullness, and continued education in the craft.

Hindu mythology wrote about nuclear proliferation 5000 years ago

I'm assuming you're speaking about https://en.wikipedia.org/wiki/Astra_(weapon):

Where would I go if I wanted to learn more deeply than wikipedia about how hindu mythology talks about nuclear proliferation? Any directions I should search?

Spiders have been a human fear for two million years, but I'm not too worried about them causing our downfall as a species. Just because we're afraid of AI doesn't mean it's actually worth fearing.

Yes, even if something is worth fearing, it's not simply because people are afraid of it already. Did something I write make you think I was arguing otherwise?

As an aside, I would argue that this deep rooted human fear of AI is related to the general ancient fear of humans being wiped out by other bands of more technologically advanced humans. Which for most of human history has been the single biggest danger by far.

People also have deep instinctual fears of things like spiders and snakes for the same reason- they were serious hazards in the environment where humans evolved.

One could say we have evolved these instincts to fear them, because they are worth fearing... not the other way around.

> Did something I write make you think I was arguing otherwise?

At the risk of having boring meta-commentary about comments, I feel like I see people often forgetting that their comments are in the context of a larger conversation and are going to be read as responses to the parent comment that they are, well, responding to. And then they get confused when their comments are interpreted in that context, rather than as completely standalone statements. So let's backtrack:

(A) AGI is a threat to human civilization. (B) Actually, its risk is marketing hype. (You) It's not marketing hype. Our fear is deep-seated.

It's not that weird to interpret the 3rd comment as disagreeing that the fear is just hype. When in fact, you were disagreeing only with the word "marketing". It's "innate" hype, not "marketing" hype. That's a much -- for lack of a better word -- weaker comment, because it is nitpicking about a minor word choice ("it's marketing hype") rather than the sentiment of the comment ("it's just hype"). So now you're asking "why did you assume my comment was interesting, and not just a little nitpick!?"

> One could say we have evolved these instincts to fear them, because they are worth fearing

But now you're back to saying AI is worth fearing, right? Isn't that what you're telling me I was wrong to assume you're saying? This feels like a needlessly difficult conversation.

All that aside, your additions to the conversation are interesting. I actually really like your take on it being a fear of technologically superior tribes. It just doesn't feel like it needs to be quite so adversarial. You established an adversarial attitude with "it's not marketing hype".

Technologically superior tribes always needed land. Their technology was advanced, but not in a way in which land is rendered useless, i.e. land price equals to zero.

Two reasons mainly for land price not equating to zero. First: buildings could not be extended upwards. Second: chloroplasts could not be produced just with air and sun.

These two problems will be solved in less than 10 years both. Land price worldwide will collapse to zero, or near zero. Buildings will be built using a flexible material, very strong and lightweight, i.e. graphene. Food will be produced using genetically modified spirulina, alongside with pharmaceutical substances and narcotics.

Also, just to reply to a parent comment, Matrix was a philosophy movie, Terminator was a comedy with some scary parts, and 2001 i didn't understand any of it. Was i supposed to be afraid of something in 2001?

I appreciate the thoughtful reply.

The seriousness of AGI risk is a big conversation our society is having right now, and I am not interested in laying out my position on it in detail on here... only that dismissing the fear of AGI as marketing hype is a straw-man argument - and ignores all of the history, discussions, and arguments on both sides of this debate going back decades, or arguably even thousands of years.

Sure, it is fair to suspect that, because I am making that point, that I might also think AGI is a real threat to human civilization. The debate is so widespread right now, with mostly aggressive and dismissive bad faith arguments on both sides, that it sounds pretty exhausting and pointless to discuss.

I share your concern, and objection to making discussions needlessly adversarial. In general, I've seen a huge rise recently in internet discussions mostly reducing to people yelling at each other that the other person is a narcissist. Even in niche technical and hobby forums, it is becoming the norm for disagreements. I feel it is infecting me as well, and it might be time to take a break from the internet and talk to people I actually like in real life more.

For sure, it's the locusts that I'm worried about.

Spiders don't, won't and couldn't control almost all aspects of human life.

This is a false equivalence.

I guess you will surely find a movie to prove you different

Couldn’t is a stretch. Microbes can control almost all aspects of human life, as evinced in 2020.

None of the movies and stories you mentioned were about the "rise of AGI". The closest you can get might be The Matrix via Animatrix? You can go look up what the authors meant by their pieces, there's plenty of analysis of each of those stories. In addition I would love a citation on "the rise of AGI" being a fear in sci-fi for a long time, I've enjoyed quite a bit of science fiction literature and I can only think of a handful that even touched on the idea.

Skynet??!!?! It seems so obvious that I am having a hard time understanding your objection... it would feel condescending to try to explain that Terminator is about AGI risk.

The modern conceptualization of AI existential risk comes largely from Vernor Vinge, in particular his essay "The Coming Technological Singularity" but it is also a central theme in most of his works, especially the Zones of Thought series.

I think it's about as much about ai risk as transformers is.

I'm terms of skynet I think you'd be referring to the third one? But that one is about drone strikes, and that's what we do today without AGI - we send out invincible killer robots to go wipe out villages from above because all males above the age of 16 are enemy combatants. The end of the movie could just as easily have been that they find a person behind the skynet controls killing everyone

The overall plot of Terminator is that an AGI called Skynet becomes self aware and unstoppable sometime in the mid 1980s and humans only hope of survival is to use time travel to destroy it before this happens. I’ll admit this isn’t very well explained to the audience, and the focus is mostly on Arnold blowing things up.

I don't think anyone is confused about Terminator's plot, but people in this thread are definitely confused about what theme means and how to do literary analysis (not surprising given the context of this website).

I understood "rise of AGI" as if preceded by "uncontrolled": a runaway intelligence that ultimately turns against its creators. In that sense, yes, those movies do include this theme.

Your comment makes sense if you intend "rise of AGI" as just growth. But then why should it be a concern?

None of those movies have AGI as a theme, you have confused "theme" with "plot point".

Terminator 2 and a deleted scene, but the essence of it was there in the movie https://www.youtube.com/v=1UZeHJyiMG8&

Corrected link: https://www.youtube.com/watch?v=1UZeHJyiMG8

Neuromancer. I Have No Mouth and I Must Scream. Erewhon.

You have massively confused "AGI as a plot point" with "is what the literary work is about". For example Neuromancer is about trauma and identity. If I'm being charitable I could give you Erewhon, but that's only because a major theme is anthropocentrism. A vast majority of the book is a criticism of victorian society.

Ellison's analysis of his own work is pretty well known and while not specifically about AGI is definitely about technology misuse, so you win there.

This is an absurd level of nitpicking... the phrase "is about" means both of those things in common speech, and everyone can tell which you mean depending on context. Language is defined by regular people through its use and context, scholars only describe it after the fact- to their extreme dismay. You might be technically correct within the jargon of, e.g. an academic literature course, but you aren't correct to insist the words mean that unless you are in that context, with that audience, where it has that meaning.

Moreover, I don't think you are correct, even in the literary sense, when saying, for example, what Neuromancer is about. I don't really care what an author says it's about, most artists are channeling something unconscious and often have very little conscious understanding of what their work is actually about. I've read plenty of Gibson books. The guy is an absolute luddite, and writes on a typewriter. He is terrified of how dangerous and dehumanizing technology is, and all of his books are "about" an all-consuming fear of technology and the changes it causes.

I don't really care about your opinions on academic literary analysis, it's OK to be wrong and have your own thoughts on the matter, but prepare to be confused when people don't agree that the Transformer movies are about the dangers of aliens turning into vehicles.

For someone who doesn't care what Gibson thinks of his own artistic works you sure do have a strong opinion on Gibson's artistic work.

You are mis-representing people that intuitively understand that a phrase can have different meanings depending on context, as fundamentally confused and actually muddling the two concepts.

Nobody would say "transformers are about the dangers of aliens turning into vehicles" but people would say it is "about aliens that can turn into vehicles." They might also say something like it's "about the the struggle to live day to day without being sure if everyday things are truly as they seem" - if that's were what it's about- I'm not actually familiar with transformers.

As an academic and a human (both!) I think it is important to remember how to keep the jargon from whatever you study distinct in your mind from common usage, so you can still communicate with people that didn't study whatever you studied.

You're welcome to make assumptions about what I'm doing, but at the end of the day you have to contend with the fact that I read a comment at face value, in the context, and then multiple people tried to back that original comment up. Maybe if you were the original author your opinion on what they were "trying to say" would matter, but it doesn't.

Who was the original author?

My son's just completed an essay on Marxist themes in Blake.

I think this guy's ideas of what literary analysis is and is not are pretty idiosyncratic, to be honest.

If not Neuromancer (although I feel differently) then Queen of Angels (Greg Bear). It's the most fascinating take on AI I've ever read - the entanglement with the themes of justice and punishment is thought-provoking.

The Turing Option (Harry Harrison, Marvin Minsky). I can't stress strongly enough, do not read this book. It's terrible. But it does exist.

Again I must stress you've confused theme with plot point. I haven't read QOA in a long time, but I'm pretty sure it was about identity and justice.

It's about an AI struggling to become conscious! Alarms go off when it uses the word "I"! The whole point of that subplot is why that AI can't achieve consciousness, but the one sent to Alpha Centauri[?] can. It writes essays meditating on its circumstances, the way humans have treated it and its sibling, it writes an essay exploring whether punishment of others is a gateway to self-awareness...

You want to tell me the theme of that thread is human identity? It's our responsiblity to our creations. It's what does it mean to be sentient? It's the ethics of using a self-aware AI as a tool. (Amongst many other things).

It contains an AI struggling to become human (to gain identity), if I remember correctly.

Either way, we're way off course from "it was a great fear of scifi that AGI would become a risk" to "a book with AGI as the vessel for metaphor about humanity exists".

Reply on news.ycombinator.com