Ask HN: What question would AI answer to demonstrate it thinks? | Hacker News Reader

Ask HN: What question would AI answer to demonstrate it thinks?

1 pointChuckMcM15 comments

There is a raging debate about LLMs and AIs in general that do not distinguish between probabilistic word selection and thought. Everyone has a different standard. My favorite (which favors not thinking) is "What are the last 10 digits of Pi?" which will give you 10 digits (typically). Another is "Describe all security vulnerabilities in the following code." I'm curious though, what are your questions? The one you would ask that, when answered accurately, would make you pause and think that maybe there was something more here than guessing the next word.

View on HN

There is nothing an LLM could say that would demonstrate thought to me. Not even close... At least with anything I can think of.

Even if you gave an super-LLM agency and a body, and it moved out to a little shack on a lake, got married, raised children, and started publishing sublime philosophy and fiction, I still wouldn't be convinced.

There's just too much human data available. Anything could be probabilistic word selection... Maybe the distinction is not really important anyway.

Unfalsifiable beliefs are always a good sign.

Well I wouldn't say that. Perhaps I could be convinced if I looked at the architecture.

But as black box (with the boundry at the text input/output), it does seem unfalsifiable.

I think a better metric than question answering will be question asking or self talk

Will it become curious or contemplate like a human?

When does it drive by itself, rather than requiring a prompt to do anything?

Well, the obvious way to do it would be to ask a question for which there is zero meaningful comment/analysis/data available online, and see how well it can truly think for itself.

The obvious problem is inventing a question like that. (You have to invent it. You can't find one online...)

Gut reaction: If all it does is sit passively and answer questions that are put to it, then it's not actually intelligent.

I genuinely think that this whole argument is a waste of time.

What matters is whether the outputs are useful and the outputs don't change based on whether you call it "thought", "AGI" or "probabilistic word selection".

At some point, they will get some basic rights, considering animal abuse is a crime, but also that we will anthropomorphize LLMs more.

Current tech is not there yet, but we should not wait to discuss either

> At some point, they will get some basic rights

I don't know how to express how deeply stupid this sounds while still being polite and constructive.

the comment you wrote adds no value

if you have nothing nice to say, say nothing at all

our ethics should lead one to believe that any intelligent entity, carbon or silicon based, deserves basic rights

Your ethics are leading you to believe this. For other people it doesn’t make sense at all that a computer program should have rights. It makes even less sense to people who know what a computer program is.

We don't give single cells rights, but we do give the complex organization of cells rights. Why should binary code never be given the same consideration as dna code? What defining factor is at play here?

To think that it is "just a program" could be like saying we are just machines without determination. This view may well become "racist" or "bigotry"

Individual ethics will determine the societal ethics that get codified into law. I have a hard time seeing how giving intelligent enough machines won't happen based on our existing ethics, laws, and history therein

> It makes even less sense to people who know what a computer program is.

I write code professionally and my beliefs are not what you claim them to be. Perhaps your opinion is the minority opinion? You should certainly not be claiming it as the de facto belief among programmers

Severance (the TV show) is a pretty entertaining exploration of this issue.

Still... Maybe its not a good analogy. LLMs are inifinitely replicable and editable. The "concious experience," if you will, is discontinuous even if you assume the architecture will advance massively. We definitely dont need to be talking about rights yet.

rights is part of ethics, which we most certainly need to be talking about

As I said, what we have today is not deserving of such considerations imho, but I do expect to see someone trying to marry an AI before I die, so this will become an issue not to far off

(in fact, someone already married an AI in Japan, and then the company that ran it closed, iirc)

They might, but they might not.

Trees have been given rights in some places. Some people believe dogs and cats have rights.

Humans have been not given rights in some places. Some people believe some humans don't have rights.

It's not about "sentience" or "consciousness" -- in reality, these concepts are religious ones, like the soul, and don't map to anything objectively meaningful.

A better way to think about rights, I think, is that they stem from the barrel of a gun. Can a thing reciprocate a social contract with me usefully? Can it help me if I'm nice to it? Can it hurt me if I'm mean to it? Alternatively, will entities that can help me do so if they see me help this thing or harm me if they see me harm this thing? That's all that matters; I'll be game-theoretically forced to grant it personhood. I'm programmed by my brain to show empathy to you, and, when that fails, you or others will harm me if I hurt you. Or, if I become a powerful dictator, I might execute you for being inconvenient to me. For all I know, you're a p-zombie that I could otherwise hurt with a clean conscience. None of that really matters; it's all about what I'm forced to do, from within or without.

Reply on news.ycombinator.com