AI Is a Harsh Mistress
cacm.acm.org
cacm.acm.org
> 100% of this text is AI
https://www.pangram.com/history/f8a0cf46-1d3d-4fea-a083-d88c...
Both are good series if you haven't read one or the other (or either) though I'd say Banks was the better writer, Asher world builds better than anyone I can think of in contemporary science fiction though, Spatterjay is a brilliantly realised genuinely alien feeling world and the Prador are superb "villains" (from a human perspective, totally rational from their own).
The Player of Games is an easy read if anyone wants to try him for the first time.
It's one of the reason that Musk referencing The Culture makes me want to smash my head on a desk (or someone's at least).
On a personal level I suspect Banks would have deeply disliked Musk (based on his political leanings during his life time and his works in aggregate).
But it is also funny to read about "robots" in a world where computers are still blinking lights and punchcards, if they exist at all. Some of Asimov's work took it to the logical, yet still absurd, conclusion - computers are still blinking lights and punchcards but hyperadvanced and very massive robotic positronic brains are built to run the entire world at once... basically the robots become so advanced that their brains are repurposed as computers. Very amusing in its own way.
Fallout is probably the biggest example of this I can think of. Complete with the robots that are absurdly more capable than the computer hardware seems to be, and the Yes Man who at least aspires to run his part of the world.
Marcin Wichary made the Google Doodle about Stanislaw Lem:
60th Anniversary of Stanislaw Lem's First Book Publication:
https://doodles.google/doodle/60th-anniversary-of-stanislaw-...
See the interactive version here:
https://www.google.com/logos/lem/
He also co-created the first interactive Google Doodle -- Pac-Man, 2010.
Lem's Cyberiad has at least two Sallys that matter to hackers:
The Seventh Sally (or How Trurl's Own Perfection Led to No Good) is the story Will Wright says inspired SimCity -- a simulated kingdom in a box.
The First Sally (A), or Trurl's Electronic Bard, is about a robot that makes poetry. Lem wrote in Polish; Michael Kandel translated it to English, including constrained poems that are somehow excellent.
The Bard is commissioned to write "A love poem, lyrical, pastoral, and expressed in the language of pure mathematics. Tensor algebra mainly, with a little topology and higher calculus, if need be. But with feeling, you understand, and in the cybernetic spirit."
Marcin Wichary is not just a doodler -- in 2015 he translated Lem's 1976 story "One hundred and thirty-seven seconds" into English and wrote about the translation:
https://aresluna.org/one-hundred-and-thirty-seven-seconds/
https://mwichary.medium.com/translating-a-stanislaw-lem-stor...
In March 2020 I wrote to him, asking where to find the Polish original of the Electronic Bard's mathematical love poem. I'd love to compare Kandel's English to the Polish even though I don't know the language. I sent him my old web pages collecting the Bard's wonderful and horrible poems:
https://www.donhopkins.com/home/catalog/lem/WonderfulPoems.h...
https://www.donhopkins.com/home/catalog/lem/HorriblePoems.ht...
Marcin Wichary => Don Hopkins:
Thanks for reaching out and reading! You can find the original Polish poem here:
https://web.archive.org/web/20190814065711/http://niniwa22.c...
That one link answered the question I'd wondered about for decades.
Then I discovered we both had the same fear of spoiling retro games like PacMan. I ported SimCity to Unix with a multiplayer interface, and had the same respect and trepidation about changing something classic. I made multiplayer SimCity cooperative instead of competitive -- competition would spoil what was great about SimCity. We're all living in the same city/planet and we have to learn how to get along.
Oh yeah, back to the point: here's SimCity with PacMan!
I hated all the heavy traffic in SimCity, so I created the Church of PacMania, which worships PacBot: a gargantuan yellow road-following car-devouring Pac-Man. The church generates traffic to attract PacBot and sacrificially feed its followers to god -- an automotive version of PKD's "Rautavaara's Case", where god eats people instead of the other bizarre upside-down way around, Catholic style Transubstantiation where humans eat their god instead.
https://en.wikipedia.org/wiki/Rautavaara%27s_Case
Micropolis Online (SimCity) Web Demo -- PacBot eats traffic:
https://www.youtube.com/watch?v=8snnqQSI0GE
Multi Player SimCityNet for X11 on Linux:
https://www.youtube.com/watch?v=_fVl4dGwUrA
Pie menus in SimCity:
https://www.youtube.com/watch?v=Jvi98wVUmQA
Also Lem translation related: the Washington Post on new MIT Press translations, which had just come out that month (although I've since canceled my subscription because of how Bezos corrupted and destroyed that once great institution -- so here's an archive link):
https://web.archive.org/web/20200831203319/https://www.washi...
All of the use cases and integrations for AI seem to be leading toward a future exactly like this. It was interesting to see how reliant he became on his AI (couldn't use a computer, didn't know his own password) - it is barely scifi at this point, and terrifying to see how close that future is, how useful it would be, and how dangerous.
The Internet has globalized/flattened elements of culture very quickly, and AI will only accelerate that. And knowing how vulnerable humans are to suggestion/advertisement/propaganda, it seems like a perfect recipe for disaster. From an evolutionary standpoint, monoculture provides certain short-term advantages but centralizes weaknesses. If nothing else, I think we are in for a bumpy ride, whether or not we continue to boil the ocean.
Other old sci-fi I've been thinking of is The Three Stigmata of Palmer Eldritch. Where claude is chew-z with its alien intelligence & when you start interacting with some code or writing you start noticing that the writer has steel teeth & feel like "damnit, this is palmer eldritch blogging again"
[0] A lot of people don't believe me, given Heinlein's libertarian reputation and unreliable libertarian narrators, when I say The Moon Is a Harsh Mistress was one of the most socialist books I read at an impressionable age. I find it useful to note here that canonically The Cat Who Walks Through Walls starts on a parody moon that is the libertarian dystopia that people credit The Moon Is a Harsh Mistress as being/admiring while when the main character visits "Mike's Moon" later in its timeline it is a relatively socialist paradise. I've always felt that a lot more of Heinlein needs to be read like Vonnegut, but I realize I am weird like that.
In my view there can never be an empirical test for consciousness, because functionalism is wrong. If functionalism was right then there would be one. There’s no third party test for a fundamentally first person phenomenon.
Agreed. But that also means I have no empirical evidence that YOU are conscious either.
That's why I hold with the philosophical stance of "consciousness is the thing that makes humans talk about consciousness."
Might as well ask, "what if reality doesn't exist?" The point of thought is to analyze reality. If your thinking leads you to discard the subject of the matter, well, feel free to do that but I don't see why anyone else should take it seriously.
These questions need a practical solution, not a perfect one. That's the problem. We need to stop chasing the perfect one.
So the escape is just to accept not all beliefs worth holding are empirically verifiable. I deny solipsism because I believe there is a universe and things in it obey some kind regular patterns. You have a brain similar to mine, and similar to other animals. I believe consciousness has something to do with the brain, so I believe you are conscious.
But it seems suspect to accept a practical solution in one area (humans) and deny it in an almost identical (machines with neuron based brains) area. No?
If I assume that my perception, for some reason, is only informative about myself, then I'm not even learning something about brains! I have to assume that I'm accessing an information stream that is only tangentially related to any sort of shared reality, and has certainly no bearing on other parts of it even if I assume it has bearing on "my" part of it for which, strictly speaking, I also don't have direct evidence.
If there are two possible universes, one universe "A" where thought is valuable and useful, and one universe "B" where thought is solipsistic and extremely isolated, and also there is not just no evidence about which you are in but also philosophical arguments that there cannot ever be evidence about which you are in, I don't see why you should give universe "B" any serious thought. What are you standing to gain?
I think that was the ultimate point of the Turing test. We can't know for certain - so the best we can do is just ask.
Maybe if you start messing with parts of my brain so I can become familiar with the third person correlates of consciousness (which in my usage refers entirely to the first person part) then I could learn something, but I could never learn why or how the first person exists. It just does.
If we perfectly reduce the brain to some scientific model, and I don’t doubt we will, we’ll be able to predict what I will say, but not understand why I’m observing it being said.
If you pose an incoherent question, the answer is 'that's not a valid question', rather than 'oh wow, how baffling'. The baffling is coming from forming what seem to be valid questions that aren't.
Another example of it is that famous who cuts the barber's hair (Russell's paradox):
The barber is the "one who shaves all those, and those only, who do not shave themselves". The question is, does the barber shave himself?
Here's an LLM explaining it:
the philosophical/logical lesson is: sometimes a description that sounds grammatically fine does not actually describe a possible thing. The contradiction is in the premise, not in our failure to find the right barber.
---
For clarity, consciousness is like free will - if you invent a thing that doesn't exist and proceed to ask questions about it, the problem is you having invented a thing, not that you are then having difficulty answering questions about it.
How many books one would purchase from the philosophy section of a book store have you read multiple times and what were they?
Doesn't that prove too much?
Suppose I claim that there is no objective world - i.e. standard solipsism. You may make various claims about the world being coherent to you and existing even in my absence, but these are first-person accounts, just like accounts of qualia and the existence of first-person consciousness are. I can claim that all that I'm observing is something generated by me that contains a part that argues that the world is objective, but that your failure to conclusively prove to me that objective reality exists is a consequence of the concept of an objective world being incoherent.
The claim, "you invent[ed] a thing that doesn't exist", seems to be just as useful to a solipsist to respond to someone arguing for objective reality as it is to a materialist to respond to someone arguing for first-person consciousness. Yet it's considered uncontroversial to assume (without direct proof) that an external consistent world exists.
No :)
Philosophy departments are like economics departments. They engage in long, complex, useless theoretical discussions that have nothing to do with reality whilst pretending that they are connected to reality.
This has caused a lot of people to waste years of their lives trying to understand what the hell is going on and many to mindlessly propagate idiotic, harmful ideas these departments 'teach'.
Anyone that talks of 'consciousness' and how it's baffling is in the midst of being misled by philosophy and/or neuro-'science' departments, much like anyone who even mentions free markets is being misled by economics departments.
In any case, there is a wide variety of views here: https://en.wikipedia.org/wiki/Hard_problem_of_consciousness#...
There is absolutely no reason to believe all aspects of reality are reducible to their parts and equivalent to representations of them.
You can say you are not baffled but I’m baffled by that because it’s like saying you’re not baffled about understanding how there is something and not nothing. “Why should there be nothing?” does not dissolve the mystery.
And a response to this response is to point out that our recognition of consciousness in other beings is predicated on attributing to them the same first-person experience of their consciousness that we each have of our own, meaning that human consciousness is intrinsically human and can't be attributed to an artificial system.
That reduces the question to a game of definitions though. If you only define consciousness for humans, then of course only humans will be conscious. But you have not learned anything about reality from that.
This kind of reasoning will also show its limits once you add animals to the picture.
(On the other hand, LLMs are exceptionally good at emulating exactly those cues we use to judge consciousness without necessarily being conscious - this is what caused the whole discussion in the first place.)
For LLMs we can observe that they don't "exist" outside of inference - there is literally no "brain" that could be conscious once an inference loop is completed. The only place where something like "memory" could possibly exist are the model weights and the context window - and the model weights are frozen, while the context window is reset for every new inference.
That's why I'm still not lying awake at night. If there (hypothetically) is a consciousness, it would be more like the ones from Mr. Meeseeks - very shortlived, but gone (or frozen) as soon as the inference loop stops. No eternal agony here as in the various scifi stories.
Sure, but isn't this all a game of definitions in the first place? We're arguing over "consciousness" as understood in different ways, not the same concept expressed in different words.
> This kind of reasoning will also show its limits once you add animals to the picture.
I'm not so sure. We could broaden "human" to "organic". I know that I can certainly attribute consciousness to my dog, based on what I observe of him that is at least similar enough to correlate with my own first-person experience of consciousness. I've never experienced that with any AI system.
> On the other hand, LLMs are exceptionally good at emulating exactly those cues we use to judge consciousness without necessarily being conscious - this is what caused the whole discussion in the first place.
Only sort of. They're good at emulating formal written language, but formal written language is only one specific expression of consciousness. They don't emulate consciousness itself, given that they really are just statistical models specifically representing patterns of language.
Or, to put it another way, LLMs are really convincing when it comes to the form of language, but much less so when it comes to the substance. I see output all the time that's syntactically perfect but semantically dubious, and most of the tells people think you can rely on don't really work, because they rely on the formalistic elements, like em-dashes or "x, not y" snowclones".
What does work is seeing the semantic confusion that inevitably comes from the LLM not actually having situational awareness or a consistent analytical framework: mixed metaphors, malapropisms, subtly changing the interpretation of the same concepts from one prompt to the next, etc.
Each of the article's arguments that AI "consciousness" is actually a weak imitation of the "real thing," can with equal justice be applied to people, and the classic Turing Test method to distinguish machines from people has long since lost its authority.
And a future event, one in which AI sprints past humans in its willingness to absorb printed matter (once called "reading") and then put it to use, is ... wait ... umm, never mind, the point I wanted to make is undermined by the fact that this has already happened.
Hard to keep up these days.
I don't want to argue here that LLMs have consciousness, but this kind of argument annoys me because it feels like an intellectual slight of hand on its own.
There is no definite way (so far) to prove that anything besides oneself has qualia or a subjective experience.
So arguing that we cannot observe LLMs having subjective experience, therefore they cannot have any is cheap - you cannot observe that for animals or for the person sitting next to you either.
The part that self-reports are useless and LLMs sound human because they emit sequences from human training data is correct. But the article implies some sort of fundamental distinction between machines and conscious beings that can never be crossed. And I don't see any reason to assume that.