Hacker News Reader

Imagine assigning every living human being a virtual AI powered con artist to tail them around and try to convince them of whatever is ordered by the highest bidder. This AI is powered by mass surveillance and big data and is able to know the "mark" intimately and work on them 24/7 both directly and via their social network. Now throw in deep fakes, attention maximizing "compulsion loops" and other adversarial models of human cognition, etc.

It's basically the hydrogen bomb of propaganda, a doomsday machine for the mind. It would be like creating Satan, except this one would be entirely amoral and mercenary.

This doomsday machine only dooms as long as there's just one user. Burger King, Subway, McDonald's, Wendy's, and Taco Bell are all going to try to hack your mind to buy lunch from their chains. But people can't eat 5 lunches for lunch. In practice this doomsday persuasion is going to bog down in a red queen's race between propagandizers/advertisers with mutually exclusive goals, much like the current media environment. Improved persuasion technology means that organizations that want to persuade people end up spending money on zero-sum games against rival organizations but public behavior in aggregate doesn't change dramatically.

Corporatism isn't the only one out there who would find such technologies useful. Besides, in this Judeo-Christian religious context there are many warnings from the Bible around such consumerism.

I am most concerned about political and nation-state uses of this technology.

Doesn't the same zero-sum dynamic apply to political persuasion? There are many competing groups in the world that already spend money on political persuasion. You said "convince them of whatever is ordered by the highest bidder" so I'm not considering the case where one secretive group with a pointed agenda (the CIA? the FSB?) has a monopoly on super-persuasion.

> I could see this ushering in the permanent eternal victory of totalitarianism since I can't imagine the masses being able to organize any resistance when constantly bombarded with propaganda from their demons.

I thought this is already what is happening now!

I think technology is neutral, but the entire system is not: it will strengthen the existing power imbalance until there is no hope of balancing it again. For example, the authority can have mass surveillance to identify possible risks and remove them, without the public noticing by controlling the social media. Even if you noticed, you have basically no chance of gathering a large enough crowd to protest due to communication channels being constantly monitored by AI. Even if you somehow get enough people, you will not have a chancw fighting them as their weapons are a lot more advanced, probably unmanned so not constrained by manpower.

That's a very bad scenario and what would make it worse for me would be a brain-computer interface that would make it impossible to tell if I was talking to a real person face to face or a person who was acting as a proxy for AI.

Without sure technology, I could take some solace in interacting with real people in real life and even if the online space was irreversibly transformed by ai propagandism. But if it reached directly into the meatsphere, then I'd be rendered hopeless. To be unsure whether I was authentically engaging with anyone or interacting with fleshpuppets proxying an AI would be to much to bear.

I cannot imagine a more complete hell. My only hope is that it is either physically impossible or economically impossible, because believing otherwise means it is inevitable.

My greatest hope is that we will somehow find a way to reduce/eliminate selfish and antisocial tendencies as an intrinsic aspect of any advances made in these areas. I think any tech which doesn't make that a top priority is doomed to create more suffering than would exist in its absence. This hope often seems like a pipe dream these days, though.

We already have mass media.

Mass media is to what I described what spears are to M-16s.

That's a pretty good analogy. In the parent post I am describing propaganda becoming mechanized warfare. One person with an M-16 could take out an entire massive army of people wielding spears (as long as they had plenty of ammo).

> One person with an M-16 could take out an entire massive army of people wielding spears

I have to disagree. All it would take is for this army to surround the person with the M-16 and wait for a magazine change, then hurl 50 spears at the hapless pincushion. And that's not counting the probability of the rifle overheating, jamming, etc. after long enough.

I realize I am taking this probably too literally, but my point stands :)

"The new feudal aristocracy would be the ones running the demons."

The question of ownership is huge and I think still underaddressed, even after some good work like the referenced Lena, or Tom Scott's "Welcome to Life: the singularity, ruined by lawyers" https://www.youtube.com/watch?v=IFe9wiDfb0E .

I was realizing a couple of weeks ago that if you do believe all the materialistic things cstross outlines, that a solid argument can be made that it may be never rational to upload your mind, on the ground that there is simply no circumstance I can imagine where you will "own" your own substrate. Arguably, owning your computational substrate is a fundamental aspect of life that we take for granted today. (Or, if you won't consider what you have now "ownership", at least nobody else does either.)

But there is no circumstance in the forseeable future in which anyone can own their substrate. For a while I thought a rich person could do it, but then I thought about the entire supply chain involved in creating any brain scanner and subsequent computational device, and the amount of hardware left out in the real world where others can affect it, and I realized, you just can never assume that you are doing anything other than flinging yourself irrevocably into a locked box that is not under your control. Even if a rich person thinks they funded the entire project, brand new software, brand new hardware, there are thousands if not millions of points where either mistakes or deliberate sabotage for control may have been done. How can they be sure? All mechanisms can be corrupted, and the interests are certainly there to do so, massively so!

Everything people do today to control you; advertising, censorship, social pressures, everything, will only be amplified and combined with new techniques if you are running on human-comprehensible hardware that can be affected by any intelligence organization, anybody in the supply chain with an axe to grind, etc.

I seriously can not think of what kind of assurances it would take to fling yourself into that. About the best you can do is hope that the math works out that it's better to keep you happy and it's not sadists pulling your strings but at least while you are working for someone else's interests they don't inflict massive amounts of pain on you just for fun. And the edited, redacted, sanitized, loyal version of yourself that happens to survive at least enjoys their servitude.

In the 1980s and 1990s, it was easy to at least imagine that maybe you'd own your own hardware. Today we can't even keep ownership and control of our mobile phones. How do you expect to have any ownership of something that makes your cell phone look like Dr. Nim? https://www.youtube.com/watch?v=9KABcmczPdg

So, there's my proposed answer to cstross: Some form of absolute right of ownership of computational substrate for all sentient beings, with the responsibility of providing that to others being absolute, even if that means some technology becomes impractical or infeasible, even at great cost. There's a prisoner's dilemma aspect to this; obviously everybody wants ownership of their own substate but the rewards for defecting can be enormous... at least in the short term.