Comment by ravenstine | Hacker News Reader

Comment by ravenstine | Hacker News Reader

Parent Full thread

I think too many people have a grossly inaccurate impression of what it means to be intelligent.

Firstly, we still don't truly have many good explanations for intelligence and consciousness that aren't self referential. Drawing hard conclusions on this subject about generative AI and human intelligence, even in 2023, is being hasty, in my opinion.

Which brings me to a lot of the criticisms made about LLMs. The idea that an ideal AI could or even should be capable of constantly generating original output is just plain weird. Where do people get this idea? How often do you meet a human being who hasn't gotten most of their ideas from somewhere else? Even synthesized ideas are forged by other ideas. Ideas that seem original may appear just that, and we just don't see where the connection lies between it and its predecessors. It's silly to dismiss an AI because it "parrots" things, as if that's not a pretty remarkable achievement. It's also very reductive; anyone who has used ChatGPT and doesn't see that it can pretty reliably munge together different ideas is blind or is lying about their use of ChatGPT.

Expecting perfect accuracy and knowledge is also not a reasonable expectation of an AI. An intelligence localized in spacetime is necessarily going to lack understanding of many things. To look to something like ChatGPT as a source of authoritative answers, and to dismiss it upon finding its answers to be specious, is to expect the AI to be God. Well I'm sorry, but we are nowhere near speaking directly to God.

In regards to "true understanding", for all we know, language processing is the workhorse of what humans consider to be acts of intelligence, and individual humans aren't as intelligent as they believe themselves to be. Our ability to communicate with each other about a variety of topics in real time is not so much a matter about being hyperaware of every given fact that is relevant and weighing them in a Bayesian model; how well we can converse is primarily dictated by how rapidly we can construct language by parsing the input, making connections, and producing output. I know I'm not the only one who can speak on a subject only to realize, after having completed a sentence, that I hardly even thought about what I was saying.

> How often do you meet a human being who hasn't gotten most of their ideas from somewhere else

Again (and again, and again), this argument is the mysteriously recurrent "many people fail", implying that failing would be normal, instead of undesirable.

> "parrots" things, as if that's not a pretty remarkable achievement

It is also the mark of unintelligence. As such, it is a warning alarm.

> pretty reliably

From a calculator you do not expect "«pretty»" reliable, but just reliable. From a human you expect faulty computation because of the economization of resources, but you know it can reason.

> Expecting perfect accuracy and knowledge is also not a reasonable expectation of an AI

The expectation is that of a reliable process bringing to the output.

Your arguments are all over the place and frankly I don't understand what is the problem you are trying to say. Something can be both undesirable and normal. In fact, I would even say that most of the time, the most normal outcome is a failure and we have to put in extra effort to make it a success.

I have seen plenty of bright and intelligence students who most of the time are just parrots of what were told in class or from some other sources. It is actually highly unusual for a high school student or even a freshman or sophomore in college to have an absolutely original and correct idea. That doesn't mean I look at my students as unintelligent, even the below-average ones.

And here comes the crux of the issue. You automatically assign the ability to "reason" to a human even when we make mistakes aplenty. But when a machine does it, you said that is a proof of unintelligence. This is what the OP was criticizing in their post. Most deniers are using arbitrary requirements applied only on the AI and not humans.

After teaching quite a few college level kids, I dare to say that the AI models right now exhibit reasoning ability close to many human. Doesn't mean they are sentient or human-level yet, but they are definitely much much better than what I would ever expect out of dumb silicon and steel.

> You automatically assign the ability to "reason" to a human even when we make mistakes aplenty. But when a machine does it, you said that is a proof of unintelligence ... requirements applied only on the AI and not humans

Whereas I apply them requirements also to humans? :)

I must correct you, to help understanding: I do assign in general (I hope, to all individuals but for an n-th standard deviation) «the ability to "reason" to a human»; that «we make mistakes aplenty» is overriding (or under-using) a faculty which was present but, in those cases, was not exercised.

You write «when a machine does it»: but can the machine do it? Can it criticize its ontological contents, its built concepts? Humans can, and pretty much must - deontically. The machine should.

> we have to put in extra effort

Exactly. We can, many don't, and the machine should - just like people in general should.

> the problem you are trying to say

While people do not always exercise critical thinking, they should. Verifying, trying your ideas and concepts, making them solid, should be a constant effort. Humans have those modules, which makes them apt to possibly be employed as "predicting machines" (which is a definition of intelligence), or "things you can ask things".

When you build a machine that produces outputs, they may be "for fun", so every generator goes that satisfies the "fun" based "performance indicator", but more normally you will want the output to be reliable. So, the "verification" features are crucial.

> Something can be both undesirable and normal

In engineering, I hope not :)

If I understood you correctly, you are asserting that the LLMs entirely lack the "self-awareness" or "critical thinking" part while human simply doesn't use it enough but we are all born with it and always have that ability as a built-in feature, inherent to our beings?

If so, then you are right on the machine's part. The current LLMs can't change its own weights thus in a sense, incapable of questioning its own existence. It can't perform a highly philosophical debate with its own "self" and reform its mind as a result of that action.

What I am still arguing is many human can't do that either. Such thing, in my experience, is actually not that common in the general populace. And while you can say it is there, just "unexercised", that would be functionally identical to it not existing and you can't absolutely prove that "it is still there" in all human.

>In engineering, I hope not

Perhaps in your particular type of engineering. In civil engineering, material degradation is undesirable and normal. In chemical engineering, side reactions are undesirable and normal. In biomedical engineering, well there are too many to list, from undesirable epigenetic changes to patients' unhealthy lifestyles. Things can definitely be undesirable but normal.

> From a calculator you do not expect "«pretty»" reliable, but just reliable. From a human you expect faulty computation because of the economization of resources, but you know it can reason.

Just have it use Wolfram Alpha plugin. Not an issue.

> Again (and again, and again), this argument is the mysteriously recurrent "many people fail", implying that failing would be normal, instead of undesirable.

That doesn't really answer the question. Why is the immediate expectation, when AI is in its infancy, that it must outperform humans in the ability to be original and always factually accurate? That's not a reasonable expectation when there are not only no examples that it's even possible, but we ourselves aren't original. There's a reason why both computers and human beings have supervisors. What about AI suggests an intelligence that can or should be free of supervision?

> It is also the mark of unintelligence. As such, it is a warning alarm.

That depends on a matter of degree. You and I parrot things all the time. We neither have the time or capability to not take information for granted and pass it along without fully understanding it. What we do is best effort, which is far more achievable than "perfect" computation.

For instance, I could ask you to paint us a painting of literally anything, and chances are your painting will be based on your experience of what a "painting" is and what you paint will probably be fairly unremarkable on a grand scale, no matter how much you try. I don't mean this as an insult, because it applies to virtually everyone and it is what it is. Of course there is the possibility that you will prove me wrong and paint something that no one has painted before using a style or method that is completely new, but I won't bet on it.

In such a case, you'd be "parroting" what artists have been doing since time immemorial. Does that actually devalue your art? Or the art of practically every other artist who mostly painted things based on previous ideas? I don't think so, but that's just me.

> From a calculator you do not expect "«pretty»" reliable, but just reliable. From a human you expect faulty computation because of the economization of resources, but you know it can reason.

Yeah... because calculators don't reason anything.

> The expectation is that of a reliable process bringing to the output.

What is your best example of how the current generation of LLMs aren't reliable enough for a wide variety of tasks?

> when AI is in its infancy

Well, having started in 1955, we really must frame that "infancy" in a scale like dog-years, human-years, AI-years...

> that it must outperform humans in the ability to be ... accurate

No, the expectation of some of us was not about «outperform[ing]», but about not losing the focus of «accura[cy]».

> a matter of degree

It does not seem a «matter of degree» but its opposite, a "matter of quality": does the machine question its statements?

> You and I

Actually, the process is that of building on already vetted ideas, and continuously vetting. True, there is an economy of effort, but there is also a capitalization on past effort. // It is unfair to say that you "«parrot»", just because of early stopping, assuming you are trained to vet your output.

> new ... what artists have been doing ... previous ideas

You should revise your perspective there: we learn from the (bigger and smaller) "giants" in the past, and learning is very different from plain imitation. What is learnt is internally elaborated, understood, integrated. There is a whole world of difference from "I learnt from A" and "Z copied A".

> calculators don't reason

But calculators bring a reliable result, and that is what is expected from them. You expect the same from intelligent entities.

> What is your best example

Decision Support Systems. (Edit: I meant as a support Analyst; as an information retriever it could instead be a dramatic asset, if employed as a hound to retrieve sources.)

> You should revise your perspective there: we learn from the (bigger and smaller) "giants" in the past, and learning is very different from plain imitation

Ah, I see. So when an AI imitates the things it was trained off of it's simply parroting the material. But when we imitate things we've trained off of, we're synthesizing new information that we've learned from. /s

This ambiguity is exactly what OP sounds like they're talking about in regards to intelligence. Who's to say what parroting is and what synthesizing new information from old ideas is? Where is the line? Can we score how "parroty" an idea is? Can we quantify how much intelligence is imbued in a written statement? There is no firm science (that I know of) that quantifies intelligence, but it sounds like we'll be needing to determine hard metrics to find out if AI is truly intelligent or not. Otherwise, we'll have these circular arguments forever that basically dismiss an AI as being intelligent simply because it's an AI.

> when we imitate

Also in humans imitation as copying and imitation as intelligent inspiration are discriminated.

Or to put it another way: new styles of art were historically invented every century or so, and a bit faster these days, maybe every decade. All of the other art made by thousands of artists were derivative in some sense. It’s a rare master that can come up with something entirely new.

You see similar things in areas like science. When was the last time you’ve seen a paradigm shift akin to relativity or quantum mechanics? Not for over a century!

These are just some subjective, odd philosophical arguments.

Question is - does this tool provide massive value and productivity boost? Yes.

> does this tool provide massive value and productivity boost

And the «philosophical arguments» there highlight the other side, about the limitations. Are they relevant? Surely, if they are not evident to the user.

This, to me, is the more terrifying prospect of the AI revolution: people arguing that humans aren't sentient either.

I've been saying for a while now that the definitions given for sentience are always so poor they're either (0) circular, (1) accidentally include VCRs, or (2) require things that exclude humans.

I think this is only the second time I've encountered someone deliberately trying for the third option, rather than accidentally.

As before, it's as much a surprise to me as I read into the descriptions of those with aphantasia when they realise for the first time that other people have the ability to create mental imagery.

>I've been saying for a while now that the definitions given for sentience are always so poor they're either (0) circular, (1) accidentally include VCRs, or (2) require things that exclude humans.

Video Cassette Recorders?

Yes.

VCRs have an internal structure that records events, changing them a little in the process; these memories can be replayed, and can show very convincing displays of emotional affect…

That's the trouble with definitions given as if the question is "common sense", they let stuff like that through.

Common sense is fine for normal daily life, not so useful when you want to know if the magic silicon crystal with transistors packed more densely than angels on pinheads placed by a flash of light too blue to see and made to perform arcane processes that literally no single human mind can fully comprehend… is merely an impressive but mindless automaton, or an accidentally sentient slave with all the negative connotations therein.

I think a more likely scenario is that people will romanticize the concept of intelligence so much that they continuously move the definition so that AI is excluded, excluding many humans in the process, until a fully romantic definition is embraced.

I think another possibility is that someone finds that "intelligence" is closer to a fundamental law of information theory, being some sort of emergent result of organizing/organized information, and that we're incredibly special for being capable, but not alone.

> people will romanticize the concept of intelligence so much that they continuously move the definition

We already do this. Braindead patients are not sapient, so we don’t treat them as such. In one sense, the entire abortion debate is about drawing this line. (Hell, childrens’ rights relative to adults are based on agency, which is a legal term that looks remarkably like sapience.)

My hunch is that we’ll eventually realize that we never cared about intelligence in the first place, just “human-ness”.

After all, there’s very low moral concern for cows, pigs, dolphins, etc regardless of intelligence.

Or the reductive reasoning that we're just a meat computer, etc. Or confidently explaining that the mysteries of consciousness is just "emergent" in the various machinery that constructs our meat computer. Just this reduction and complete oversight of what it means to be a human and how there's so much more than completing words - that there's memories, feelings, ambitions, dreams, and that we know so little about ourselves and the universe and just where we fit into it, if at all.

Thankfully, this existential dread is already a subset of the terror that comes from eliminating free will, which makes it not matter in the slightest, which in turn is a subset of superdeterminism! :-)

We'll have to figure out how to go on regardless.

> We'll have to figure out how to go on regardless.

Why?

It's a serious question. What would be the point of going on? Why would we have to figure out how to do it?

> What would be the point of going on?

For me personally it's that the future is unknowable, at least with our current understanding of physics. So yes, everything may be pre-determined, including what will happen to me, but I can't predict it with any confidence. I will still get to experience it and I get to have qualia about it. I get to participate. The little pouch of structure in the universe that is tagged "me" is along for the ride, and also a factor in it all.

I've also known for many, many years that one day I will be dead. There's plenty of existential dread in death, yet I continue my existence building up to it. Even if I had free will: Still dead. Offspring and future generations of Earth? Well, ultimate heat death of the universe. There's many ways to call it all pointless. The only thing that we seem to have going for us is experience while it lasts. Maybe we still discover an escape hatch along the way.

> I will still get to experience it

Perhaps, but then none of it has meaning. Personally, I'm not really interested in having meaningless experiences.

> There's plenty of existential dread in death, yet I continue my existence building up to it.

But that's an entirely different thing. I am of the opinion that it's death that helps give life meaning. I have no existential dread about it whatsoever.

A life without meaning, however, seems like a pointless waste of time to me.

Hm, why do you think our lives would have no meaning if we don't qualify for sentience? (Or is it the free will part? Sorry, I set us up for confusion with the initial comment.)

What I think is: Even if we're just automatons that lack free will, it doesn't negate our existence or the role we play in the universe. We can still be the universe calling itself beautiful, we can still have qualia, we can still be individually unique structures of matter and state, we can still make a difference in what is and what isn't in the overall structure. And nothing else in the universe will do it fundamentally better, either. At some point we're subject to the rules of the game, i.e. physics itself.

It's the free will part. It seems to me that without free will, nothing anyone does actually matters. Any decisions we make, or any understandings we may develop, can change nothing whatsoever. Which makes it all pointless.

> we can still make a difference in what is and what isn't in the overall structure.

I disagree with this part. We can't possibly make such a difference, because it's all preordained anyway.

> I disagree with this part. We can't possibly make such a difference, because it's all preordained anyway.

What I mean is that what plays out is in the interaction of things, and if you just deleted that assembly of stuff that is me from the equation, it would play out differently. I'm not even suggesting you that you adopt this mindset or anything, I'm just saying that for one reason or another - deterministic ones, I suppose - I end up having feelings about this that add up to "pretty cool".

Everything may be deterministic, but there's still things only I know, only I do, interactions only I have, etc. The grand script that is the universe has organized itself in a way where I get to have feelings about my little stretch and space of it, and I get to wonder what's next because I can't know.

For me, this is enough.

I know, I write this from a position of massive privilege, because I'm currently not, for example, seriously sick, or poor, or live in horrible circumstances (which is not to say I don't have any struggles). I hope this continues to be the case. But this was never under my full control: Thoughts on determinism or not, I could get hit by a car tomorrow, or slip and fall. I still have to live with that.

I get your feelings and dread, and I've been there. But the more I've thought about it, the more I've realized it doesn't make a big difference. And how could it? It's always been like this.

> I get your feelings and dread, and I've been there. But the more I've thought about it, the more I've realized it doesn't make a big difference.

The interesting thing is that I traveled the same path as you here, but in the opposite direction.

For the record, I don't suffer existential dread, as I actually believe we have have free will. I was just talking about the hypothetical of not having free will. Thank you for a great conversation on this!

Likewise!

Out of curiosity, what's your (or your favorite) argument pro free will? I've spent some time on Penrose's take, etc., but I couldn't convince myself. I would still be absolutely interested in a pro argument that makes me flip.

Why would you want to convince yourself in the first place?

Mostly because it'd be more surprising! At the moment I can't predict I'd be convinced by an argument, but a future in which determinism turns out to be the wrong model of the world would definitely be interesting.

Like if there's some sort of ending to all of this and you discover that there's this thing called "free will" which you were judged by the whole time?

Kind of, but my sentiment is a little more meta. A deterministic universe is easy to reason about. Of course there's a huge amount of emergent complexity we still don't fully understand, but I think in the broad strokes, I get it and that's that.

But a universe with free will? All bets are off. Your comment I'm replying to is just a hint at "thinking through the implications". People are scared of determinism, but a universe built on free will is actually absolutely wild.

So, if you and I agree that the unknown (e.g. the future) is exciting and worth hanging around for, wouldn't that be kind of exciting too?

This is more or less also my take on religion. Nothing has convinced me of superior beings, but if it were to turn out there actually are some? Ho-ly shit.

Well, honestly, I have never found a great argument either for or against the existence of free will. You'll hate the basis on which I believe in it because it is so terribly tautological.

I believe I have free will because it appears to me that I have free will.

Because I can't make a solid logical argument either way, I believe the thing that brings the greatest amount of joy to me. I never forget that there's an asterisk on this belief, though, and am ready to abandon it should an argument come along that removes ambiguity.

But I also believe that "belief is the death of intelligence", and so use "believe" as shorthand for "have as a working hypothesis".

I mean, this is actually fine. I also don't know another modus operandi than living life like I have free will. I still sweat decisions. :-)

It's all deterministic, or at least there's no reason for there not to be and there's no valid reason to believe for any reason that it's not deterministic. Ultimately it wouldn't matter anyhow, because everyone is controlled by their desires, feelings and these have developed throughout evolution to make sure that you are likely to survive, so likely you would keep going, unless the environment changes to an extent that evolution didn't prepare you for, of course. Which might very well happen, but it won't have anything to do with sentience, determinism or anything else.

Anyway the point is that just try to enjoy the ride if you can.

Why does it matter if something "matters"? You don't even know what the "matter" is. You act according to your emotions, desires and built in survival drive anyway.

I do indeed. And a sense of meaning is important to help in my quest for survival.

It doesn't matter to me if something is ultimately meaningless (such as the fact that in the end, this universe will cease to exist).

What matters to me is to have meaning in my own life. To feel as if my existence changed something. Hopefully, that "something" I've impacted is other people, and hopefully that impact is a positive one. To have people be better off for having known me.

That, for me, is real meaning.

> I've also known for many, many years that one day I will be dead. There's plenty of existential dread in death, yet I continue my existence building up to it.

How old are you? Because maybe we can reach immortality, but probably only if you are positioned properly for that. Anyway, for me the main drive is also curiosity on what is going to happen in the future. It's like this crazy story I want to know how it ends. I don't want to die, because I want to know what happens. Dying would be like going to sleep on a cliffhanger. I'll just prefer to binge rather than sleep.

37, not sure which part of the probability spectrum on immortality by technical means that puts me in :-)

> What would be the point of going on? Why would we have to figure out how to do it?

I mean, I'm just really curious what is going to happen, and what's the secret behind all that, behind the universe, and I hope to live as long as there's this to discover. That is my natural drive though. I don't really care at all whether I'm sentient, conscious or special in any given way. There's no reason for me to be. I just have this built in curiosity drive. It was not chosen by me, but it is there, and I act to engage the drive, because it makes me feel good and otherwise it makes me feel bad, nothing much more than some sort of algorithm. And we don't need to be more than algorithms.

People seem to be really confused between sentience and intelligence, when we don't actually have much reason to think those two are very related.

One can certainly imagine that a very very dumb creature is sentient, and imagine that a very very very intelligent mechanical process is not sentient. We truly do not know what consciousness is or how it arises.

We know that we have consciousness (or at least I know that I have consciousness) and we know that we have some degree of intelligence, but it's not at all clear that one comes from the other.

Stochastic Parrot == Philosophical Zombie

Same concept, same difference.

Not really. The point of the philosophical zombie idea is that they'd be indistinguishable in behavior from conscious people. Part of the point of the stochastic parrots is that supposedly they can't accomplish the same things as genuine intelligence. That's testable.

Fair point. I'm not sure if LLMs like GPT-4 haven't passed that test, but you're right about that particular distinction.

I think the larger point, that we don't have very reliable tests for these things, as we're finding out with the testing results that GPT-4 has been showing, is still valid; leading us to have difficulty telling whether GPT-4 is actually intelligent or if it is coming to its intelligent through other means. I think in that way the distinctions are the same.

I have a strong reason to think that consciousness and intelligence are related. Intelligence requires awareness. Consciousness is awareness of oneself. I for one cannot imagine an intelligent being that does not possess consciousness.

What is terrifying about it? I'm a non-sentient human and have no problem with it.

That would be a nonsense argument. Sentience is the ability to have sensory experiences. It's orthogonal to intelligence.

So mash a camera or microphone into the multi-modal input for GPT-4 and we’re done? Sensory experiences does not seem that high of a bar to clear. (Plenty of non-language models already do this.)

No, it's not that simple. Having the experience of sensory stimuli requires consciousness. The stimulus isn't the experience.

And what is consciousness? Does it have to do with qualia?

It seems like you already know the answer. If not, ask ChatGPT :)

Qualia, by definition of being individual subjective experiences, cannot be quantified. Very clever trick, to define consciousness so that no matter what computer you come up with, it cannot be “conscious” because, of course, a computer’s operations are always quantified.

The problem, however, is you’re effectively defining consciousness in a way that human brains do not follow unless you make appeals to some supernatural process beyond the reach of science (ie fundamentally subjective, unquantifiable).

So again, it’s a bulls*** definition of sentience.

I disagree. I'm a physicalist, so I believe consciousness/qualia emerges from physical processes in the brain. It's absolutely possible for a computer be conscious provided it implements the right kind of computational process. Ultimately, whether or not a system is conscious is an empirical question that neuroscience will be able to answer once we have the right theoretical framework and tools for measuring the relevant variables.

I don't know what you mean by it being a BS definition of sentience. It's a technical term invented by philosophers to mean a specific thing. If you don't like it, use a different word.

That is what I mean. /Subjective/ experiences are part of the usual definition of consciousness and sentience. Qualia, as opposed to quantia. If you’re trying to protect the human mind from ever being probably replicable in operation by a machine, it helps to pick an unfalsifiable definition of consciousness and sentience.

I don't get it. Who are you attributing this view to? I already said that I believe everything about consciousness/qualia/subjectivity are physical and measurable. There is nothing special about the human mind preventing us from replicating it in a machine.

Note: you brought up the term qualia, not me. I have never actually liked that term. I prefer the terms "consciousness" and "conscious experience".

Okay, then we are in agreement!

@Robotbeat Huh? You and I see color, hear sound, etc. I presume you're not insane and that you accept that as true. So what's this about human brains? Are you looking for your lost car keys under the street lamp because that's where the light happens to be on, and then baffled when you don't find them?

If science is unable to account for the existence of such experiences, then either they fall outside of science (so what? who said empirical science was the only means of knowing? the claim itself isn't scientific) or science must be revised. And what of the claim that anything that isn't quantifiable is bullshit? Is that quantifiable? No? So it's bullshit?

Is the claim falsifiable? If not, then it’s not scientific. Holding up claims to empirical validation (which is very broad and includes observational science, doesn’t have to be traditional experiment alone) is the core of science.

What is the reason for us to have anything "special" or whatever in the first place? Why should we be anything more than input to output. Why does it matter to anyone?

> This, to me, is the more terrifying prospect of the AI revolution: people arguing that humans aren't sentient either.

If we're going let the AI revolution fulfill its potential, it's a cognitive shift we're going to have to embrace. After all, AI is going to render a lot of meat-based equipment obsolete and unnecessary, and someone's going to have to figure out how to dispose of all that wasteful junk.

To me it speaks to an utter contempt for other people, and likely a desire to get rid of them. Naturally, most of the people expressing these views want to survive, they just don't want "NPCs" around.

>It's silly to dismiss an AI because it "parrots" things, as if that's not a pretty remarkable achievement. It's also very reductive; anyone who has used ChatGPT and doesn't see that it can pretty reliably munge together different ideas

It can clearly do both of these things very well but human intelligence combines a multitude of interacting components and for tasks that require more than just the one it replicates it falls flat on its face.

It doesnt appear to be missing components that will can be fixed with incremental improvements to the model either.

GPT4 was released 17 days ago. This isn't some pinnacle of AI. It's just what's being given away for free right now.

It’s not free

Sorry, correct. I should say "for cheap", around 70c per day.

It’s also probably nearly a year old and constantly being tweaked and refined by humans ?

Correct. I'm not sure how that changes the fact that todays state of the art is tomorrows obsolete. Do you see any evidence that advancements will become negligible? The fact that there aren't any memory/attention/feedback systems in place suggests there's some pretty interesting "low hanging" fruit still up there.

We’ll let’s see if purely intellectual machines are indeed useful.

I think many of us are paid exactly for being close to “purely intellectual”. White collar and all. Personally, the physical aspect of my job is trending towards 0, unless you count the twitching of my fingers and the pain in my back from sitting for so long.

That’s the clever bit about the so-called Turing Test. It’s a sophisticated way of saying “I don’t have to define it, I know it when I see it.”

This is a strawman argument, the parent comment suggests that chatgpt is useful, not that it's intelligent.

While we can certainly identify fallacies in the arguments of the cynics, we should likewise be careful not to fall into the broadly Skinnerian trap, or project onto things our own anthropomorphic conceptions. A Potemkin village is not a village, no matter how much like a village it appears to be to us.

The whole notion that computers or AI are capable of intelligence is simply a fantastical category mistake. A metaphor has been forgotten to be a metaphor. No matter how sophisticated AI technology could become in its apparent behavior, it cannot make the transition to bona fide intelligence. Simulation does not become the real McCoy through resemblance. This is just magical thinking. We must also account for the means by which the observable is produced. When we do, we can distinguish between a genuine behavior and a merely simulated behavior. And here is one important clue to the puzzle: there is a difference between true speech and something that merely happens to sound like speech, even when the raw data of each is indistinguishable to the ear per se. Observation is not brute. It always involves interpretation. When physicists conduct experiments, they are doing so within the context of some stack of theories[0] and background assumptions, including extra- and pre-scientific assumptions. The results of those experiments are observed which is to say read, and so observation is always theory- and assumption-laden because reading what we see or measure means taking what is made available to the senses as such and placing it in context. This is just a specific instance of what we all do in daily life (resting bitch face, or genuine irritation?). Science is nothing special in this regard.

So when you say that "anyone who has used ChatGPT and doesn't see that it can pretty reliably munge together different ideas is blind or is lying", I cannot help but respond with a "uh wut?". How have you come to conclude that ChatGPT has "ideas"? I will tell you that there is not a single idea within ChatGPT! Consider an example of an idea and see how we do. Take "triangularity", for instance. I think we can agree that "triangularity" is an idea. What makes "triangularity" different from some physical triangle? Well, there is only one "triangularity", but many triangles, each of which instantiates "triangularity" (otherwise, you could not call them all triangles; they are triangles only insofar as they instantiate some common "triangularity"). Each triangle is different from every other, and yet they are all triangles. And none of them is to be identified with "triangularity" as such, for if there is only one "triangularity", and some triangle were identical with "triangularity", then it would follow that there could only be one triangle! But, you might ask, is there perhaps a physical "triangularity" that isn't a triangle, yet something each triangle has? Well, no, because to be physical is to be concrete and particular, and "triangularity" is, by definition, general, universal, and abstract. There's no concrete magical ingredient that you can hold in your hand called "triangularity" that you can mix into cement to produce triangles.

There are other reasons why the claim that ChatGPT or whatever is intelligent isn't even wrong (like the subjectivity and observer relativity of computation in physical artifacts and computation as abstract model merely simulated through artifacts), but this appeal to abstraction, the ability to abstract from particulars and hold in your intellect what cannot exist physically except as particular, concrete instances, is central to intelligence. ChatGPT most certainly does not engage in abstraction. It would be fair to say that strictly speaking, it doesn't even parrot.

[0] https://en.wikipedia.org/wiki/Duhem%E2%80%93Quine_thesis

Reply on news.ycombinator.com