Hacker News Reader

Comment by Sol- | Hacker News Reader

Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI? Perhaps all their tech is pre Butlerian Jihad, but even that seems questionable.

Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.

Additionally, the transhumanist ambitions represented by the Bene Gesserit or Navigator seem on par with all AI dangers to me. Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?

View on HN

>Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI?

This implies their technology is unfathomably complex, too complex for the human mind to understand. Doesn't seem to be the case.

>Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.

Ships are using Navigators, who are trained in "higher mathematics" and are also prescient. They have no need for AIs, nor do they wish to court destruction if they are found out (I can only imagine the BG will have the Spacing Guild by the balls if they knew).

> Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?

The books address these questions, especially the one about a certain God-Emperor...

The Bene Gesserit tried to solve alignment through their breeding schedule, but Lady Jessica threw a wrench into it by bearing a son (Paul) out of love for Duke Leto. Paul wasn't aligned with the Bene Gesserit, and hilarity ensued.

So I agree, I don't see why thinking machines are a worse alignment problem than enhanced humans. See also the Wrath of Khan.

I try not to think too deeply about it, or I won't enjoy the story. How could a society have gravity suspensors, "shields", and lasguns, but no advanced computing capability somewhere? And why use swords/knives for fighting, when projectile weapons are so much more deadly? I still love the books, but I have friends who can't enjoy the Dune world due to its many contradictions.

Edit: I realize that "shields" supposedly made ordinary projectile weapons ineffective - but they had advanced drone technology (e.g. the assassin drone used against Paul, which was also stupid)

They had advanced computing, a whole region was full of it: IX.

The rest of the empire was too feudal and too religious to dabble into advanced technology. And the other players (the BG, the Tleilalxu, the Guild) were against it as well.

What good are drones, except as weapons of assassination? Can't use them in an conventional war, there are shields. And they're very expensive, remember, the Great Houses don't have advanced manufacturing capabilities (that's IX). And, finally, there are no great wars in the empire, most houses keep to their planetary fief; the Baron's attack on the Atreides' forces on Dune was an exception. It was also mindbogglingly expensive, 80 years of spice profit.

>What good are drones, except as weapons of assassination? Can't use them in an conventional war, there are shields.

Presumably a drone could get close enough to push a blade through the shield (maybe use a rocket booster or some other advanced "suspensor" tech) at low-ish speed (a knife would have to travel through the shield on the order of a human reaction time, otherwise even knives wouldn't work. That's about 0.1s/0.1 m, so 1 m/s - ish.)

if it moves slow enough to pass a shield, it moves slow enough to be easily dispatched. And remember, no autonomous behaviour, there has to be human operator. And it's very expensive. It just doesn't scale.

OK we are really nerding out here but I just can't resist :)

The propelled blade weapon doesn't need a control system in the "shape of a human mind" to be effective. A very simple control system would work. If they have ornithopters, then they have control systems. The blades could even work like hawk talons (so an ornithopter drone falling on a shielded fighter, then springing what is effectively a pointed metal trap.)

But more importantly, you are right that if the weapon moves too slowly, then you could just get out of the way. That's true of any weapon, no matter how it is propelled. That's why I said above you have to be able to penetrate a shield faster than a human reaction time, otherwise no weapon would be effective.

You're right about the suspension of disbelief necessary to enjoy the story, of course. I just felt like nitpicking because people seem to take the Butlerian Jihad seriously as a possible way to contend with AI in our real world, which I think asks too much of the story.

> Paul wasn't aligned with the Bene Gesserit, and hilarity ensued.

Hah, great perspective and I agree. Even normal humans are so difficult to align, no wonder it went wrong with a guy with prescient powers.

Humans come vaguely prealigned due to whatever is encoded in genes and also due to limitations of human bodies that put important constraints on individuals (e.g. no infinitely copyable trusted subagents). Even if you made them superhuman in some aspects a lot of that would still remain. It seems unlikely that minds constructed by a different process would end up humanlike because they lack the evolutionary path-dependencies that shaped humans. Current models appear somewhat human due to imitation learning/pretraining, but A) this could be deceptive as we don't know what's going on inside B) history has shown that that imitation learning becomes unnecessary once RL becomes good enough (e.g. AlphaGo -> AlphaZero), meaning we might end up with minds created from random initialization.

Why are you assuming that some technology can only be discovered by AGI?

"Who solved the alignment problem for these superhumans?"

The gun, pointed to their head.

Reply on news.ycombinator.com