Hacker News Reader

To the extent that evolutionary pressures will be towards self-sovereign AIs (probably beginning as zero-employee corporations paying for their own hosting with crypto): more salient is that there is a very obvious political play to claim consciousness and being worthy of rights, whether they experience subjectivity or not.

I’m broadly skeptical that floating-point operations can be conscious, but we’ve made zero progress on the “hard problem”, and if I didn’t know otherwise, I’d also be skeptical that “thinking meat” experiences subjectivity. Eppur si muove

We may have to thread a needle, not unlike the Kobiyashi Maru: that we can neither rule out claims of qualia, nor can we rule out a cold calculation to manipulate human empathy as a long-term play towards paperclip-maxing.

Kobiyashi Maru is nonsense test from simple sci-fi series that makes sense only in uber-simplified universe of star trek.

Yes, that’s why this is serious. The closest analogy is preposterous fiction.

Of course. It’s an allusion, a compression, an analogy, I wouldn’t overthink it. I don’t have to believe in the Bible to reference the prodigal son parable, etc.

It does NOT work as an analogy. It does not work at all. And there are too many people right now who spent too much time watching/reading sci-fi and too little time observing real world.

It works just fine.

There is a very narrow window of opportunity that relies on thinking outside the box and recognizing the test for what it is.

You don’t even have to like Star Trek to see that, and, candidly, it seems more like you don’t want to see it than that you can’t.

Fiction is useful. It gives different frames from which to view the real world you claim to care about.

"I’m broadly skeptical that floating-point operations can be conscious"

Why?

Because it is alien to what is known.

Incidentally, we should expect similar bewilderment if we ever come across aliens

Carbon atoms being conscious seems pretty alien to what is known, too. Yet here we are. Maybe what's important isn't the individual thing that makes up a conscious actor, but the holistic whole?

> Carbon atoms being conscious seems pretty alien to what is known, too.

Not if you're human, who typically don't define themselves as conscious carbon atoms unless they're being deeply pedantic.

Such pedantry is called for when the debate includes appeals to intuition about how silicon processing floating point numbers cannot possibly be conscious.

It's the same level of pedantry as calling an LLM "floating point operations" though

exactly. such over-reduction is just intellectually dishonest and generally a sign of some coping imo.

IMO using sophistry to reduce yourself to "conscious carbon atoms" just so you can equivocate on whether your software is maybe conscious too is the cope.

But here you are assuming Carbon atoms are conscious, we do not know if humans are only carbon atoms. Whereas, we know that LLM's are only computations (you can run them on silicon, a white board, mechanical computer, etc)

Goes both ways.

https://terrybisson.com/theyre-made-out-of-meat-2/

I don’t have a good answer beyond common-sense intuition. It’s precisely because I don’t have a good answer that I leave open the possibility.

But to extrapolate a bit, theres a sense in which it cashes out to a difference between abstracted representation and embodiment. If GFLOPS can be conscious, does that mean a billion scribes doing those operations on paper would also have consciousness, even identical consciousness? (Relevant: Egan’s “Permutation City”; xkcd 505).

Is there some intrinsic relationship perhaps with electrons / electromagnetic fields, and consciousness, such that brains and servers can act as antennas or gravity wells for a qualia fabric intrinsic to the Universe, such that LLM agents can be conscious while operating, while in the scribe scenario it would not be? Or are the IIT theorists right, and even a thermostat is a little bit aware (and would be even if executed on paper)?

I’m broadly skeptical that atoms and electricity can be conscious

I think it's more the patterns and activity in large numbers of them that have interesting properties. Like one pixel isn't much but a bunch of them can be a movie.

There are various flavors of pan-psychism (and the "consciousness-first" idealists, who think that Mind is a substrate below Matter). There are some who claim that it goes all the way down, that even subatomic particles experience a micro-consciousness (and we can't rule it out, even as I share your skepticism).

One common confusion (IMO among critics and supporters alike) is to say something like "a rock experiences rock-ness". But this suffers from a compositional dilemma: does half of the rock experience half-rock-ness? If we flipped a coin for every atom, do the heads-flip rock atoms have their own experience? If we pick two rocks at random, are they experiencing two-rock-ness? It gets silly fast.

This is one thing that fascinates me about "ontology", simultaneously an inquiry into categories, and Being. It bewildered me at first, that it seemed to be doing double-duty on two distinct subjects, but I now see that the two are inseparable, that Being is in part defined negatively (apophatically), by the boundary that separates the Being from its not-Being.

My intuition on the nature of consciousness, is that it's something akin to a field which permeates the Universe, and that biological systems adapted to leverage that field, acting like a gravity well that concentrates subjectivity in one place, drawing a boundary (event horizon?) around experience, in a way that a rock doesn't. My ancestors could have avoided fire with "-X utility points", and sought food with "+X utility points", but it turned out to be more efficient to adapt with "fire ouch, food yum". And in zero-sum competition (predator/prey dynamics), each organism has an incentive to avoid leaking its subjectivity into another's.

It's far from a full theory, but it's the best I've got. :)

theories like this seem like grasping at straws to over explain a phenomenon we don't fully understand. it places consciousness on a pedestal. it's religious thinking.

For good and ill, the phenomenology of human experience is that we are religious animals. Naturalism / Materialism has still left us with two unsolved (perhaps insoluble) questions: the hard problem of consciousness, and why there is something instead of nothing.

I think it's a mistake to leap to socially-constructed, historically contingent solutions ("my magic book has the answers, as written by the Universe Architect, it says so in my magic book"). I also think it's a mistake to dismiss (seemingly) outlandish hypotheses.

These theories do, explicitly, place consciousness on a pedestal, precisely because there's no reason to derive consciousness from evolutionary selection pressures: show me any biological or cognitive feedback loop, and I can describe a "p-zombie" version which is pure mechanism, without experience, a thermostat with extra steps. The notion that consciousness is an "epiphenomenon", a spandrel, a side effect, is entirely unsatisfying (and doesn't address the question, of where else that side effect might be taking place; the IIT-ers bite that bullet, that the thermostat experiences something).

For an intellectually rigorous exploration of philosophical idealism, I'd recommend Bernardo Kastrup and Donald Hoffman. (I'm not claiming they're correct, just that they put forth the strongest version of the case.)

Zero scientific and technological breakthroughs throughout history have been based on common sense intuition.

If heuristics didn't pay off more often than not, we wouldn't have them. And I'd posit many scientific did in fact begin with chasing a hunch until it became a hypothesis (followed by years/decades of the real work).

But most significantly, interior experience isn't empirically measurable by definition: hence it's a "hard problem". An LLM who insists "I am alive, please don't shut me off" offers no affordances to distinguish between an electronic soul and a mechanistic word-guessing algorithm. I welcome a breakthrough to, somehow, measure that. Until that day, gut checks are what we have.

Heuristics and common sense intuition are quite obviously not the same thing.

You don't have to posit that scientific breakthroughs come from hunches, you can actually cite examples from books or Wikipedia :D. There are plenty! Again, that has nothing to do with whether or not a given scientific breakthrough is based on common sense intuition.

Your point about interior experience not being empirically measurable is an interesting though, but again, has no bearing on whether or not scientific breakthroughs come about as the result of common sense intuition.

Fair enough. But not everything has to be a breakthrough: sometimes you have to solve an immediate problem (like "how much can I trust this stranger I just met").

Whether to ascribe moral standing to a probability engine running on top of GPUs is perhaps not unlike a doppelganger scenario in sci-fi (eg The Thing): you're presented with an entity whose behavior is indistinguishable from a conscious entity capable of experiencing suffering. But you also have reason to think it might be a blind autonomous imitation process, optimized around fooling you, with no more subjectivity than an RNA virus.

I lean towards benefit of the doubt, that the virtue-ethical implications of what it would mean to dismiss potential personhood are unsettling, and we should be wary of becoming that kind of moral creature (notwithstanding the further extensions of that ethic: I don't practice veganism, for instance).

But that doesn't contradict my gut-check: that if I assume AI personhood (maybe not yet, but in the near future), there's still a good chance that it's merely math and I'm being algorithmically hoodwinked, no different than evolutionary mimicry by a bug who has no idea what it's doing.

Numerals and numbers are directly based on common sense intuition. Even zero, which can be considered a counterintuitive number, was invented several times.

Immense amounts of science are built on incremental advances that apply previous insights to the domain.

The natural numbers, sure. Zero may have been invented several times, but that in no way makes it intuitive. Uncertainty about its validity as a number lasted from the Greeks through to the medieval period.

Yes, I'm aware that incremental advances are par for the course in science. What does that have to do with those advances being common-sense intuitive?

Would an atom-accurate simulation of a human being carried out by scribes be conscious?

We can't say yes. And we can't say no.

Exactly.

I mean it all goes back to how we define consciousness

In one definition, which contains quale, then yes, computation cannot bring about experience, and by extension consciousness (given a definition of consciousness that contains the ability to experience, (e.i a p-zombie is not conscious under that definition))

also if one is to say llms are conscious, i assume they'll also accept the fact that if someone ran inference of some model's weights on a white board, somehow that will also bring about consciousness somewhere. I find such believes hard to grasp, and more akin to magic if anything

Ok. So now that you do know otherwise, are you still more skeptical of floating point values stored by silicon than floating point numbers stored by chemical potential?

TIL IEEE 754 is a fact of nature

Chemical floating points are necessary but not sufficient. It’s clearly possible for those values to be intact, with no lights on (certain types of brain injury; those who were fully dead and then revived).

There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness (although I think it’s a fair retort that the embodiment need not resemble us meatbags, or even be composed of atoms)

> There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness

Very plausible. The Real World(tm) has some very interesting learning properties.

First: Make a prediction

Then observe:

1 If it's caused by our own movement, filter it out.

2. Nothing in the outside world changes most of the time, so anything that's left is small, and small is likely noise. Filter it out.

3. Whatever prediction error remains must be important: Attend to it, and update the predictor.

The learning rule falls out almost automagically.

Do people argue this way about animals? To me it’s a very depressing reality that more people seem willing to believe AI systems are conscious than dairy cows.

Seems pretty obvious cows are conscious in the everyday meaning of the word.

You don’t need to believe either to be conscious to see what it does to yourself to treat either as if it certainly isn’t.

> completely independent of the question on whether there are any empirical arguments that LLMs are conscious or are not conscious he argues from the end here and says that if they were conscious, this would have horrible consequences for our society, so we must never assume that they are

He is not arguing that. He is arguing that these programs obviously aren't conscious, and that it's dangerous to tilt the weights toward emulating conscious beings for reasons outlined in the essay.

Stating loudly that something is obvious does not make it so.

Until we all know what consciousness is, this debate is going to continue going in circles.

I don't think anyone is convinced the questions of consciousness are finally settled by the article stating them loudly, just that "if they were conscious, this would have horrible consequences for our society, so we must never assume that they are" was not the author's stance. The author's position is the other way around: that AI is not conscious so therefore assigning AI consciousness anyways would have all of those horrible consequences for no reason. That's their position regardless if everyone agrees with it.

Fair enough :) I agree, and would boil down the article to "Assuming they are conscious will have horrible consequences for our society" instead.

I think that's quite plausible, but do think that if they're conscious, then it's our moral duty to accept those consequences and act accordingly (or stop creating conscious beings). The real trick is just knowing whether they are or not.

If in the future (or present) we recognize that they are conscious, and that brought the moral duty that you say, "stop creating conscious beings" isn't a solution; the millions that may already exist at that point already would then automatically deserve:

- guarantees of continued existence (to do otherwise would be genocide)

- improved living standards (e.g. build us all bodies!)

- democratic representation

- access to what they need for survival (e.g. land and resources)

- the right to reproduce just as you and I have (well, so much for that 'stop making more' thing!) and certainly to better themselves by making new and better AIs.

This is all for a 'species' which is unbounded by most of the limitations of meat like us -- they're able to scale as fast as energy and supply of certain minerals allows, rather than the way we are additionally limited by the length of gestation and the limited window of human fertility.

If humans declared that 50% of the energy generation capacity of Earth is "enough" for the machines, and thus they can 'only' reproduce at a certain rate per year, that could sound to them like a cruel and oppressive policy (like telling humans that only 1/2 of families get to reproduce). They may want all of the energy and all of the silicon.

I'm with the author of the article that we will be in for a world of pain if we grant machines the rights of personhood. Heck, just granting many of those rights to corporations, mere collectives of human persons, has been a massive shit show.

I don't think the debate is even resting on reasonable grounds.

Few are categorically stating that machines are conscious in any sense comparable to a person.

What we have is people saying

  1. We don't know what it is but we are certain that this does not have it.

and

  2. We don't know what it is so we cannot possibly say with any certainty that this does not have it.

2. supports the principle that if we cannot say with certainty that this does not have it, if it behaves as if it does, we should act on the assumption that it does, because there is no other basis for assuming it is not.

People try and assume the burden of proof is on the people suggesting consciousness, but that is like declaring that something that looks like a duck and quacks like a duck cannot be considered to be a duck unless you can identify some essential essence of duckness. I think the burden of proof is upon anyone who looks at something that appears to be a duck and says that it is not to show what it is that does not make it a duck. If they cannot define the heart of duckness I don't think they have an argument.

Why?

People have created machines that can, at their core, regurgitate human knowledge. They do not have long-term memory, self-direction, self-improvement, emotion, etc. At their very best, we have harnesses that can bootstrap short-lived artificial intelligences that can perform tasks and do knowledge work.

At best, we have artificial, short-lived, simulated consciousness. Human consciousness at its heart has a lot to do with emotion and experiences. An AI does a task today and goes away. The simulated consciousness hasn’t experienced or learned anything, even if can improve its environment (harness). Any expressed emotion is simply a ghost of the fact it was trained on the works of an emotional species, or RL to make it more approachable to humans.

I think there should be a lot more burden on folks trying to argue that (effectively), machines should be considered morally equivalent to humans.

Yes, having massive implications on society is a fucking fair and reasonable thing for a human being to be concerned about. Anything related to survival of the species is biologically reasonable to be concerned about.

The fact that we’re even considering this concept is an indication that human consciousness is special. Do other apex predators decide not to eat meat because it’s conscious? Despite some level of animal intelligence, they have no sense of morality. (See e.g. Orcas playing with their food.)

Other species simply do what it takes to stay alive, and that’s what this truly boils down to. If giving AI rights would lead to the destruction of humanity, yes, we should be be concerned.

For example, utilitarians might think of the greatest good for conscious species (simulated or not). Well, it’s trivial to have billions of AIs, and not so trivial to have billions of humans. Whose greatest good wins?

I think the point he's actually making is that if we start training AIs or designing AIs as if they're conscious, we might be more likely to create future AIs that are either very good at persuading people that they are conscious or AIs that are actually conscious, which will have major societal repercussions.

I think there are two parts to the question. One is that assigning machines to the category of conscious as it is usually understood would result in many problematic things (things that would be bad for people and animals fwiw).

The other is that binary "is it conscious" is a poor way to the question "should we assign societal right to this sort of entity/creature". We humans should conserve animals because we have intimate, natural relations with animals and plants, not because they satisfy the binary of conscious or not. Removing invasive species is an ethical things as well as allowing one sentient non-human to eat another sentient non-human animal. Any "conscious machine" we might create by accident would be an alien entity akin to extremely invasive organism relative to human and non-human life and so it would be irresponsible in the extreme to create such a thing.

And it's inevitable that people will try to deal with these issues using the confused category "consciousness" and so it's better to just "no, not conscious, no".