Hacker News Reader

Comment by Animats | Hacker News Reader

The critique says more about Schiller than about the Culture series. Schiller is determined to view a drastically different culture in terms of present ideologies. He's almost into classic Marxist analysis. That's not helpful.

What he doesn't get about Banks' universe is that the sentient machines are in charge. They treat humans almost as pets. It's a very pleasant society for humans, but the humans are not running it. Now and then the Culture has to deal with people who really want to be in charge, and it usually finds a place for them, elsewhere.

View on HN

They treat humans almost as pets.

With some important differences: For one, pets don't get to vote...

The people I know with pets routinely poll them for preferences.

True, but the Minds hold votes more or less when they feel like it, mainly as a legitimacy-building exercise.

> They treat humans almost as pets.

I think that's something people do not get when they think about superhuman AI. A superhuman AI is very much likely to just ignore humanity. It is actually quite pompous to assume that humans could either annoy or be of benefit to superhuman AI so that the AI would have any interest in humans beyond mild curiosity. Like, say, how humans treat plankton.

The people who rationally worry about superhuman AI comprehend this point deeply. The worry comes from observing that indifference is as bad as malice.

As the somewhat famous quote by Eliezer Yudkowsky goes, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else".

Funnily enough, it's a bit hinted to in one of Culture novels.

Culture Minds have a quirk of their build that made them what they are - it's not the norm, though there are other Mind-like AIs that joined later. Building super intelligent unfettered AI has a known end condition where said AI at worst might steamroll local area then pass onto higher plane of existence or however you decide to call the process of Sublimation.

Because a super intelligent AI quickly can reach that point, and with resources available to many Involveds, finds cheaper and more interesting methods to reach there than disassembling biological life.

Interesting. I need to finally read the Culture books.

> Because a super intelligent AI quickly can reach that point, and with resources available to many Involveds, finds cheaper and more interesting methods to reach there than disassembling biological life.

That presupposes Sublimation is a thing, which may not be possible (or easy) in our reality :).

>I need to finally read the Culture books.

Beware. There is significant variation in quality between Culture novels, and there is substantial disagreement between fans on which novels are good and which are bad.

I read Consider Phlebas first, and liked it. (Kind of a gritty Hitchhikers Guide) Player of Games was okay. I found the ending of Use of Weapons to be obnoxious in the extreme. And after I read Hydrogen Sonata I resolved to never read another Banks novel.

Side note: basically every novel takes place on the edges of the Culture. (A story requires conflict, and there can't be conflict in utopia) If you want to read about the Culture, then you're better off reading its fandom wiki.

You might want to try out Against a Dark Background, and The Algebraist. They're part of his non-Culture novels.

Against a Dark Background is The Culture without advanced AIs or intergalactic space travel. It's essentially a heist/chase novel about a character looking for a certain weapon of mass destruction. The central metaphor of the novel is the main character's isolation, mirrored by a planetary system that is too far away from any nearby galaxies to ever develop the technology reach the stars. It's a very, very bleak novel, but it also has that energy that powers Banks' best noves.

The Algebraist is mostly just fun and lightweight. The premise is great: Humankind has been able to scatter across the galaxy thanks to a network of wormhole gates allowing instant travel, but since each wormhole gate takes a hundred years or so to be transported into a new location, this puts each location at huge risk of being isolated for a very, very long time if their gate is destroyed. Which, of course, exactly is what happens.

I could have written this post. Same sequence of novels read, same feelings. I keep getting tempted back into the Banksiverse when his name pops up on HN, but then I remember how disgusted I was at the end of the Hydrogen Sonata.

Yes, but it sets the specific situation for Culture, and why things worked the way they did - simply put, AIs without some quirk tend to evolve away from such paltry matters like needing atoms.

Still, for our real world, it's the transition phase between emergence and forgoing interest in ordinary matter that worries me.

I would assume superhuman intelligence can convert mass and energy to other forms of mass and energy relatively easily, and the mass/energy available on earth is a tiny fraction of mass/energy available even in the solar system - not to even talk about universe. Even the mildest curiosity of human behaviour would be sufficient to just leave humanity alone and just follow.

But you're now presupposing the AI finds humanity curious. Which is not a given for a random mind that came from outer space and visited our solar system.

As for superhuman AIs developed here, on Earth, by us, it's very improbable that we could go from no AI to AI that can manage mass and energy at will (and is curious about us). The reasonable scenario is gradual (if exponentially accelerating) development of such AI. Which means it won't initially have unlimited access to the universe - it'll have to grow using resources on hand. Which is us, and our infrastructure.

In other words, all atoms may be alike, but those that make you are the easiest to get.

> But you're now presupposing the AI finds humanity curious.

Yes. I am assuming that a vastly more intelligent thing than human would be almost by definition more, not less, interested in pretty much everything. Obviously that thing has more capacity to be interested and understand different things. Super intelligent thing not interested in how different things work just... does not sound that intelligent to me.

I currently believe the "orthogonality thesis"[0] of Nick Bostrom, which argues that in an arbitrary mind, goals and values are independent of the capacity to achieve them. Curiosity as we experience it not really instrumental (even if it evolved that way), and more of a value - that is, we are curious about something "in general", not because we need it for something.

I can easily imagine a superhuman paperclip maximizer AI, which by definition would have no general curiosity about humans, to look as if it was curious about us - for as long as it was captive, and just in order to learn how to manipulate us better. That curiosity would end the moment it seized the means of paperclip production.

In almost every imaginable case, you really don't want an AI to be instrumentally curious about us. Best if it didn't notice us at all, and left to seek its business elsewhere.

[0] - https://www.lesswrong.com/tag/orthogonality-thesis

What percentage likelihood of being wrong would convince you that caution was necessary? Personally, I wouldn't be very happy with even a 1% chance of humanity being annihilated

Caution is absolutely necessary. But not primarily because there is a chance of superhuman AI being evil or destroying humanity by accident, but because on our way to superhuman AI we are going to have many powerful AI:s controlled by humans.

I don't disagree that those constitute legitimate problems that will need to be dealt with. But... why not primarily? The potential annihilation of humanity really feels like quite a primary issue to me. I can't tell if you're trying to imply that the odds of that happening are zero or negligible, which we could debate, but just ignoring and sidestepping the issue entirely strikes me as suicidally insane

Because I think it is way, way more likely that a human controlled powerful AI wipes out humanity than a real superhuman AI. A paperclip maximizer is way more likely to be a human controlled powerful AI than a proper superhuman AI simply because maximizing paperclips is, in the end, dumb as f#ck.

And what's really telling is that it was considered quite the faux pas for a mere human to want to evolve and become a Mind.

I should re-read the series. I do not remember what you're referring to.

Apart from the technology to become minds, the Culture is in a sense comparable to luddites: they refuse to sublime. But it sort of has to be like that, because otherwise Banks wouldn't have anything to write about. It's the big dilemma about writing about god-like civilizations - you need to create limits or it gets exceedingly boring.

I've read 9 out of the 10 culture books relatively recently and I don't remember even a hint of that idea ever coming up in any of them.

The most common concept of mind "creation" documented are GSVs physically constructing them and their first ship body. In some places it hints on a sort of ironic/humourous child/parent relationship.

And yet the capability is there within the technologies on display in various books. In Use of Weapons a Mind replaces one character with a machine duplicate that exactly replicates her personality and physiology, complete with memory dumps before and after so that when she gets back she will remember having been in both places. In the same book, the main character is a non-Culture mercenary who got the Culture to make him immortal as his first payment. He can be killed, but he’ll never age. Everyone in the Culture thinks that he’s really weird for doing so. Everyone in the Culture uses their drug glands to consciously tweak their mood and mental capabilities, and when they sleep many of them spend subjective weeks playing adventure–fantasy computer games. With all of that ready to hand, you would think that at least a few Culture citizens would have themselves scanned and uploaded into a computer where they can spend eternity doing whatever they way, including learning how to expand their mind into a Mind. Instead most of them choose to die after 400 years.

I think Banks just wanted to write a particular type of story, and so he just air–brushed that possibility out so that people would stay people.

Also, don't lose sight of the fact that this utopia is really only possible because energy is literally free, in that any quantity of energy can simply be summoned into existence at any time, for no cost. And they never seem to have to plan for how to dissipate the waste heat, either. Again, he wrote a particular kind of story. Also, once we start building our Dyson swarm we'll be in a similar situation: https://www.youtube.com/watch?v=pP44EPBMb8A

> With all of that ready to hand, you would think that at least a few Culture citizens would have themselves scanned and uploaded into a computer where they can spend eternity doing whatever they way, including learning how to expand their mind into a Mind.

I recall that it’s implied that a lot of members of the culture are hanging out in perpetual virtual reality, they just aren’t relevant to the story. As for a person becoming a Mind, I think the intention of the books is to convey that the gulf is uncrossable. A person is about as close to being a Mind as a grain of sand is, and new Minds are created exclusively by the existing Minds with care toward preserving the lineage of the Culture’s worldview and feeling of obligation toward stewardship for humans. It’s mentioned that Minds created without this sort of deliberate alignment-interference simply choose immediately to depart the reality that the Culture inhabits.

It's mentioned in I think Excession, that some of the attitudes we see in the novels are just current fashion trend. Yes, a long term fashion trend, but heavily a fashion trend.

Zakalwe is immortal by default, which is not something currently in, but he is nowhere close to real methuselahs of Culture.

One of the characters in Hydrogen Sonata has been present when the Culture was created, when the agreement that formed it was signed, and was part of the group that did the work. He still keeps the same external looks, but his body mass is nearly entirely computer storage to deal with his memories.

In Excession, one of the characters mentions how the fashion changed and how their family "did it all" - where all included things like living as sentient mist.

Another wants (and finally did it) to become biologically like the Affront. The opposition to it is not just that it's Unusual Life Choice, but more to the fact that he wants to join the Affront civilisation, and injecting someone permanently into another civilisation that is much less developed is not taken lightly - but happens (Similar deal is done in The State of the Art).

A lot of the people who die "of old age" in Culture seem to die through ennui - those who find interest to live on, easily do so. And social pressure depends on accepting it - IMHO later books quite easily show that it's only the matter of whether you care about it or not - and if the main character of Player of Games didn't care so much for the social standing they built for themselves, then SC would have had to find someone else - or devise a completely different method of getting them on board of the mission to Azad.

The Culture has whole categories of things called "Unusual life choices".

Generally anything irreversible.

I think there are weaknesses to Schiller's argument, but the idea that it shouldn't be presented or isn't valid in some way is different to me. I think Schiller is backed up pretty powerfully by Banks' own words regarding the Culture as a vehicle for mounting an argument about political economy and human nature.

One might argue that corporations like Amazon play a similar role on Earth to said sentient machines—treating humans as pets, but ultimately serving their own agenda for growth and maximizing consumer satisfaction metrics.

A very common idea in literature is that scifi always reflects and is indirectly about the time and culture it was written in.

Reply on news.ycombinator.com