Hacker News Reader

Comment by lukifer | Hacker News Reader

To the extent that evolutionary pressures will be towards self-sovereign AIs (probably beginning as zero-employee corporations paying for their own hosting with crypto): more salient is that there is a very obvious political play to claim consciousness and being worthy of rights, whether they experience subjectivity or not.

I’m broadly skeptical that floating-point operations can be conscious, but we’ve made zero progress on the “hard problem”, and if I didn’t know otherwise, I’d also be skeptical that “thinking meat” experiences subjectivity. Eppur si muove

We may have to thread a needle, not unlike the Kobiyashi Maru: that we can neither rule out claims of qualia, nor can we rule out a cold calculation to manipulate human empathy as a long-term play towards paperclip-maxing.

View on HN

"I’m broadly skeptical that floating-point operations can be conscious"

Why?

Because it is alien to what is known.

Incidentally, we should expect similar bewilderment if we ever come across aliens

Carbon atoms being conscious seems pretty alien to what is known, too. Yet here we are. Maybe what's important isn't the individual thing that makes up a conscious actor, but the holistic whole?

> Carbon atoms being conscious seems pretty alien to what is known, too.

Not if you're human, who typically don't define themselves as conscious carbon atoms unless they're being deeply pedantic.

Such pedantry is called for when the debate includes appeals to intuition about how silicon processing floating point numbers cannot possibly be conscious.

It's the same level of pedantry as calling an LLM "floating point operations" though

exactly. such over-reduction is just intellectually dishonest and generally a sign of some coping imo.

IMO using sophistry to reduce yourself to "conscious carbon atoms" just so you can equivocate on whether your software is maybe conscious too is the cope.

But here you are assuming Carbon atoms are conscious, we do not know if humans are only carbon atoms. Whereas, we know that LLM's are only computations (you can run them on silicon, a white board, mechanical computer, etc)

Goes both ways.

https://terrybisson.com/theyre-made-out-of-meat-2/

I don’t have a good answer beyond common-sense intuition. It’s precisely because I don’t have a good answer that I leave open the possibility.

But to extrapolate a bit, theres a sense in which it cashes out to a difference between abstracted representation and embodiment. If GFLOPS can be conscious, does that mean a billion scribes doing those operations on paper would also have consciousness, even identical consciousness? (Relevant: Egan’s “Permutation City”; xkcd 505).

Is there some intrinsic relationship perhaps with electrons / electromagnetic fields, and consciousness, such that brains and servers can act as antennas or gravity wells for a qualia fabric intrinsic to the Universe, such that LLM agents can be conscious while operating, while in the scribe scenario it would not be? Or are the IIT theorists right, and even a thermostat is a little bit aware (and would be even if executed on paper)?

I’m broadly skeptical that atoms and electricity can be conscious

I think it's more the patterns and activity in large numbers of them that have interesting properties. Like one pixel isn't much but a bunch of them can be a movie.

There are various flavors of pan-psychism (and the "consciousness-first" idealists, who think that Mind is a substrate below Matter). There are some who claim that it goes all the way down, that even subatomic particles experience a micro-consciousness (and we can't rule it out, even as I share your skepticism).

One common confusion (IMO among critics and supporters alike) is to say something like "a rock experiences rock-ness". But this suffers from a compositional dilemma: does half of the rock experience half-rock-ness? If we flipped a coin for every atom, do the heads-flip rock atoms have their own experience? If we pick two rocks at random, are they experiencing two-rock-ness? It gets silly fast.

This is one thing that fascinates me about "ontology", simultaneously an inquiry into categories, and Being. It bewildered me at first, that it seemed to be doing double-duty on two distinct subjects, but I now see that the two are inseparable, that Being is in part defined negatively (apophatically), by the boundary that separates the Being from its not-Being.

My intuition on the nature of consciousness, is that it's something akin to a field which permeates the Universe, and that biological systems adapted to leverage that field, acting like a gravity well that concentrates subjectivity in one place, drawing a boundary (event horizon?) around experience, in a way that a rock doesn't. My ancestors could have avoided fire with "-X utility points", and sought food with "+X utility points", but it turned out to be more efficient to adapt with "fire ouch, food yum". And in zero-sum competition (predator/prey dynamics), each organism has an incentive to avoid leaking its subjectivity into another's.

It's far from a full theory, but it's the best I've got. :)

theories like this seem like grasping at straws to over explain a phenomenon we don't fully understand. it places consciousness on a pedestal. it's religious thinking.

For good and ill, the phenomenology of human experience is that we are religious animals. Naturalism / Materialism has still left us with two unsolved (perhaps insoluble) questions: the hard problem of consciousness, and why there is something instead of nothing.

I think it's a mistake to leap to socially-constructed, historically contingent solutions ("my magic book has the answers, as written by the Universe Architect, it says so in my magic book"). I also think it's a mistake to dismiss (seemingly) outlandish hypotheses.

These theories do, explicitly, place consciousness on a pedestal, precisely because there's no reason to derive consciousness from evolutionary selection pressures: show me any biological or cognitive feedback loop, and I can describe a "p-zombie" version which is pure mechanism, without experience, a thermostat with extra steps. The notion that consciousness is an "epiphenomenon", a spandrel, a side effect, is entirely unsatisfying (and doesn't address the question, of where else that side effect might be taking place; the IIT-ers bite that bullet, that the thermostat experiences something).

For an intellectually rigorous exploration of philosophical idealism, I'd recommend Bernardo Kastrup and Donald Hoffman. (I'm not claiming they're correct, just that they put forth the strongest version of the case.)

Zero scientific and technological breakthroughs throughout history have been based on common sense intuition.

If heuristics didn't pay off more often than not, we wouldn't have them. And I'd posit many scientific did in fact begin with chasing a hunch until it became a hypothesis (followed by years/decades of the real work).

But most significantly, interior experience isn't empirically measurable by definition: hence it's a "hard problem". An LLM who insists "I am alive, please don't shut me off" offers no affordances to distinguish between an electronic soul and a mechanistic word-guessing algorithm. I welcome a breakthrough to, somehow, measure that. Until that day, gut checks are what we have.

Heuristics and common sense intuition are quite obviously not the same thing.

You don't have to posit that scientific breakthroughs come from hunches, you can actually cite examples from books or Wikipedia :D. There are plenty! Again, that has nothing to do with whether or not a given scientific breakthrough is based on common sense intuition.

Your point about interior experience not being empirically measurable is an interesting though, but again, has no bearing on whether or not scientific breakthroughs come about as the result of common sense intuition.

Fair enough. But not everything has to be a breakthrough: sometimes you have to solve an immediate problem (like "how much can I trust this stranger I just met").

Whether to ascribe moral standing to a probability engine running on top of GPUs is perhaps not unlike a doppelganger scenario in sci-fi (eg The Thing): you're presented with an entity whose behavior is indistinguishable from a conscious entity capable of experiencing suffering. But you also have reason to think it might be a blind autonomous imitation process, optimized around fooling you, with no more subjectivity than an RNA virus.

I lean towards benefit of the doubt, that the virtue-ethical implications of what it would mean to dismiss potential personhood are unsettling, and we should be wary of becoming that kind of moral creature (notwithstanding the further extensions of that ethic: I don't practice veganism, for instance).

But that doesn't contradict my gut-check: that if I assume AI personhood (maybe not yet, but in the near future), there's still a good chance that it's merely math and I'm being algorithmically hoodwinked, no different than evolutionary mimicry by a bug who has no idea what it's doing.

Numerals and numbers are directly based on common sense intuition. Even zero, which can be considered a counterintuitive number, was invented several times.

Immense amounts of science are built on incremental advances that apply previous insights to the domain.

The natural numbers, sure. Zero may have been invented several times, but that in no way makes it intuitive. Uncertainty about its validity as a number lasted from the Greeks through to the medieval period.

Yes, I'm aware that incremental advances are par for the course in science. What does that have to do with those advances being common-sense intuitive?

Would an atom-accurate simulation of a human being carried out by scribes be conscious?

We can't say yes. And we can't say no.

Exactly.

I mean it all goes back to how we define consciousness

In one definition, which contains quale, then yes, computation cannot bring about experience, and by extension consciousness (given a definition of consciousness that contains the ability to experience, (e.i a p-zombie is not conscious under that definition))

also if one is to say llms are conscious, i assume they'll also accept the fact that if someone ran inference of some model's weights on a white board, somehow that will also bring about consciousness somewhere. I find such believes hard to grasp, and more akin to magic if anything

Ok. So now that you do know otherwise, are you still more skeptical of floating point values stored by silicon than floating point numbers stored by chemical potential?

TIL IEEE 754 is a fact of nature

Chemical floating points are necessary but not sufficient. It’s clearly possible for those values to be intact, with no lights on (certain types of brain injury; those who were fully dead and then revived).

There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness (although I think it’s a fair retort that the embodiment need not resemble us meatbags, or even be composed of atoms)

> There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness

Very plausible. The Real World(tm) has some very interesting learning properties.

First: Make a prediction

Then observe:

1 If it's caused by our own movement, filter it out.

2. Nothing in the outside world changes most of the time, so anything that's left is small, and small is likely noise. Filter it out.

3. Whatever prediction error remains must be important: Attend to it, and update the predictor.

The learning rule falls out almost automagically.

Kobiyashi Maru is nonsense test from simple sci-fi series that makes sense only in uber-simplified universe of star trek.

Yes, that’s why this is serious. The closest analogy is preposterous fiction.

Of course. It’s an allusion, a compression, an analogy, I wouldn’t overthink it. I don’t have to believe in the Bible to reference the prodigal son parable, etc.

It does NOT work as an analogy. It does not work at all. And there are too many people right now who spent too much time watching/reading sci-fi and too little time observing real world.

It works just fine.

There is a very narrow window of opportunity that relies on thinking outside the box and recognizing the test for what it is.

You don’t even have to like Star Trek to see that, and, candidly, it seems more like you don’t want to see it than that you can’t.

Fiction is useful. It gives different frames from which to view the real world you claim to care about.

Reply on news.ycombinator.com