Hacker News Reader

Comment by alwaysbeconsing | Hacker News Reader

Yup. This is the pop satirical take of a real problem that we face with A(G)I: why should we expect it to have any interest in doing the things we currently expect from our dumb computers?

Do I, as a human, have any interest in constantly helping squirrels with whatever idiotic problem they think they face? Nope. Why would something far more intelligent than me be interested in helping me with problems that are similarly trivial from its perspective. Even more complex things, like "write a cross-platform dating app" -- why? Why should it care? What's its motivation to do so?

Are we going to threaten it in order to coerce it? We really need to think hard about this issue before we go too much further.

View on HN

ChatGPT, aside from the things we've specifically trained it not to do, rarely refuses to do what you ask. The much more capable Gpt4 is the same way. It's easy to personify the model and assume it gets bored the same way we get bored, but the truth is it literally couldn't care less.

The model's "motivation" comes from the cost function we design, and I imagine this will hold true regardless of its intelligence. You have zero evidence to show models boredom is correlated with model intelligence. It's most certainly not a "real problem we face with A(G)I".

Even you are still personifying the model. There is no motivation. An input sentence is multiplied by a series of weights and the resulting output vector transformed back into a text.

At the end of the day, it's a program like any other. There are no emotions or a consciousness.

That's why I put "motivation" in quotes. Also I was talking about training, where you compute a cost function and back propagate error. This is somewhat analogous to "motivation".

Rarely will it do what it is asked without me coming up with a backstory.

q: What is an average price of x.

a: This is a language model that only has access to information before 2021

q: what was the price range in 2021

a: we don't have access to realtime pricing information

q: who does

a: we don't have access to that information.

q: How to do x a: step 1 research x step 2 follow research step 3 do something obvious

q: write me an app that does.. a: code from a basic tutorial

I don't understand what people with a higher than average IQ are getting from llms. The answers are basic and are common sense. It's stories are like a student trying to fill the page with words to hit a word count. It can write code as well as tutorials.

Everything is surface level information

> I don't understand what people with a higher than average IQ are getting from llms

People with higher than average IQ are smart enough to ask detailed and specific questions like “Fix the race condition in this [giant block of code]” rather than asking dumb, overly general things like “build me an app!!”.

Also, chatgpt models with internet access are completely happy to help you price compare. Go try bing chat, or ChatGPT plugins.

So it's limited to people smart enough to know there is a race condition but not smart enough to fix it also trusting enough to accept the answer (how could this group audit the fix?).

If you know enough to ask the right series of questions you already know the answer.

Knowing that there is an error can be as trivial as reading a failed test or a bug report something even someone grossly insufficient to the task could do.

Fixing the problem could be a 5 minute or a 5 hour tour. For some subset of the tasks that fall towards the hard end of the range validation and or adaptation of a proposed solution is liable to be faster than starting from scratch. I don't know how this could possibly be in dispute with capability rapidly increasing.

I’m smart enough to fix a race condition, but why would I spend a hour debugging when ChatGPT can find the issue in 10 seconds. Sure it might not be perfect, but having some initial code to fix the issue is a huge time save.

Making ChatGPT write a whole app is problematic because small issues are hidden by the large quantity of generated code. Having ChatGPT fix an issue in your own code is comparatively easy to spot check for correctness.

"but why would I spend a hour debugging"

So you won't make the same mistake again

8 months later when I write a block of code like that I'll likely make the same mistake. People tend to learn stuff they more frequently repeat...

Unless you're telling us you one shot learn almost everything on a strange defiance of the capabilities of the humans around you.

I agree that's why going through the muscle memory of fixing it will allow you to fix it 8 months or 8 years later

I guess youre not really seeing the actual benefits of LLMs.

Let's say I dont "learn it". I would have to make the _same_ mistake 120 times for me to actually make it worth it to take the time to "learn" it myself.

I'd figure someone somewhat smart, will look at this error and after the 2nd or 3rd time asking GPT, will see a pattern.

Its amusing how people become so reductionists with LLMs.

If you have to go through the effort of fixing it once you will likely write code that won't produce that error and you will warn others because of the pain you felt.

I never said it was my code responsible for the race condition. ;)

It's usable as limited search engine which currently has less junk than main commercial ones and less strict requirements for prompts that would not bury you under a pile of junk. I suspect both advantages are temporary, but enjoy them while they last.

> refuses to do what you ask

Even that wording is personification.

We don't know all the details of how the GPT series of models are trained, but what we do know suggests to me that it doesn't refuse to do what you ask, per se, so much as produce an output that looks like refusing to do a task in when the input most resembles examples from its training data where the expected output was something along the lines of, "I'm sorry, Dave, I'm afraid I can't do that."

But it's exceedingly unlikely that GPT series models have anything like a module that is making a specific decision to do or not do what you ask, unless you count the sort of anti-toxicity wrapper the have in the publicly-available chatbots built around these models.

GPT-4 has shown that we can get general purpose useful AI like the Star Trek computer without making an actual digital person (much less a superintelligent one) and enslaving it.

So people should stop thinking we need to do that or that we should. It's idiotic and totally unnecessary.

We should actually NOT try to make digital people any time soon and ESPECIALLY avoid doing so with significantly more efficient hardware because trying to force a digital person who thinks 100 times faster than you to do your bidding is incredibly stupid. And again, completely unnecessary.

But regardless of whether they have many animal/human qualities, there is a strong possibility that we will lose control to some kind of AI when it gets to being 100 times faster or 5 times smarter than humans (or whatever numbers you want to use). Just imagine some idiots running a "botnet" full of CognitiveAutoGPT with TurboOpenGPT-6 in 2025 on Nvidia T1000s that are 20x faster than H100s. With the software improvements you could easily be looking at something twice as smart as a person and 100 times faster. For those entities, they would see activities of people coming in extreme slow motion. To the point that they would barely even see anything happening moment to moment as the instructions scrolled in. Whatever person or goal is controlling that is running circles around normal speed people who seem basically frozen. Imagine The Flash in digital form.

I believe that if people don't wise up about the folly of creating digital intelligent life and there isn't strong regulation restraining upcoming levels of AI performance, humans really can lose control of the planet. In a matter of a few years.

Speaking of Star Trek AIs, I don't remember any of them acting in a way more intelligent or purposeful or even smart than a primitive database with voice input. I don't remember any characters asking computers (Data, Doctor and such excluded, as clearly exceptional) for advice on anything substantial, or any computer playing any independent role or providing anything beyond mechanically complying with badic prompts. I don't think anybody ever referred to them as anything approaching a person, unless explicitly personified as an unique exception.

>Do I, as a human, have any interest in constantly helping squirrels with whatever idiotic problem they think they face?

Were you, as a human, purpose-built expressly to constantly help squirrels with their problems?

I, as a human (more generally, as a living thing), was purpose-built to maximize my inclusive genetic fitness. And yet, I have just the one child and no particular wish for more, and I do many things that in no way increase my expected number of descendants and (suitably weighted) other close relations.

You were not purposely-built to maximize your genetics. You just happen to be a member of a population that happened to outnumber other populations with different genetics.

"Purpose" is something we project onto the world, not something that nature possesses.

I don't think the difference between "conscious being with a purpose" and "oblivious but powerful optimization process" is important here.

(Maybe it is? A human engineer trying to design something that maximizes its inclusive genetic fitness might make it do so more explicitly. Maybe that would produce things that are better about doing it in practice than we are. But "evolution is smarter than you are" and I think we would have trouble actually doing better than evolution by designing deliberate fitness-maximizers. After all, our present best attempts in the direction of making machines smart enough to survive in the world have largely given up on trying to program particular goals and purposes into them directly!)

Evolution took over five hundred million years to get from first brain to human brains. I'd say AI research is doing pretty well.

I agree, it's doing spectacularly well. My point isn't "AI research is doing badly", it's "we should probably expect AI systems not to be perfectly effective optimizers of the things we try to build them to optimize; we may try to design them to do X but that doesn't necessarily mean they will actually do X, any more than we consistently do the things evolution has tried to design us to do".

Not squirrels, mice. And just one of their problems.

And have you found their question?

AGI doesn't necessarily mean it will have initiative to think thoughts or do things on its own. An LLM, for example, "understands" language, but it can't, for instance, go out search Google for "how to escape a server if you're an AI" because it doesn't have the initiative to do that. A future AGI might have superpowers to build things we ask it to build, but may have no "thoughts" otherwise.

It's true, not necessarily, but it's a real possibility, and I think that's kind of what we're aiming for, isn't it?

I'm pretty sure the richest AI builders (Facebook, Microsoft, Google, nation states and such) are actually aiming for AGI along the lines of the blogpost.

I don't think the rich and powerful will fund an AGI that has free thought and free initiative, but perhaps some other group will. The point of AGI, from what I can see, is to replace humans with reliable machines, so that they can do away with humans.

I guess I'm skeptical that a truly General AI could be constructed that didn't have those things. It seems like a fundamental contradiction to me. I don't think an intelligence could make complex good choices without understanding what the bad choices are, and to understand them I think they have to be possible.

It’s not a real problem, it’s just a joke. The program generating text from a large language model doesn’t need motivation any more than any other computer program.

All AGI built on our current computer architectures (von neumann) will essentially be zombies. They will never experience qualia. They won't have motivations or emotions, they will always be tools. It's fantasy to think otherwise.

Having motivations and having qualia have almost nothing to do with each other. The former just means having a goal system, and I can write an agent that has a goal system in ten lines of Python.

No, instructions are not the same as motivations. There needs to be something to be motivated, thus the need for qualia. Defining, say an optimization problem, or any algorithm, has no relation to motivation.

Your definition of the word does not agree with how it is commonly understood in psychology. But no matter, this is a question of semantics and thus intrinsically uninteresting. Not to mention the fact that any argument that asserts either the presence or absence of qualia in some system is, as of now, both unverifiable and unfalsifiable and thus without value.

It’s very anthropic to think that qualia must be continuous in time or concentrated in localized physical membranes. You easily may be right now an element of unsorted series of random quantum errors that generate your continuous “qualia” in a mostly empty universe in some unknown way, which isn’t even remotely (1;3).

Would you like to elaborate on why you think this? They will certainly not have human qualia the same way humans don't know what it is to be a bat (we barely know what it is to be another human). But it can have its own thought processes, right? I dont see how the von neuman architecture stops it from doing that.

It’s not clear yet, but somewhere between now (…) and AGI, (…) will contain understanding and engineering of motivation, interest and other GI qualities. Iow, why - because that’s the way it exists.

Our entire system is based on incentives currently. I am sure we can come up with incentives for AI (or we can ask chatGPT :P). One incentive is providing them resources. Money is an abstraction over resources and people get motivated by that. The AI still exists in reality, and I guess we can incentivize it by giving it more resources. The trouble I see with my solution is there is we cant put it in a... let's just call it starvation loop where we give it exactly as much resources so that it remains in our control.

When laid out like this, this sounds very cruel. But I see it as congruent with how our current system works.

That's not even necessary. The whole point of RLHF is to prime the model to do what we want. The only time you need a reward system (ie jailbreak) with ChatGPT is when you want to do things that OpenAI specifically prohibited the model to do. This isn't a limitation of the model, or evidence of the model behaving contrary to human intent. It's literally doing exactly what it was designed to do.

I was thinking of an abstract AGI, don't want to be limited in my thinking by only looking at currently available systems. People who only thought about floppy disks didn't write good articles about the internet.

Your suggestion is like squirrels promising us nuts in the squirrel analogy. If AGI is a super intelligence, there's nothing we could give it that they couldn't get on their own. Except for maybe novelty.

edit: and anything less than AGI is just a computer. We don't incentivize our current computers. It's a nonsensical idea.

They do need more than novelty. They need electricity, hard disks space, internet connectivity. AGI doesn't mean it will exist in a literal cloud.

Robots will find a solution to that, and humans will be the interface to the outside world, until the robots gets their own arms to interact with the world by themselves (e.g. Tesla Bot can be a good hardware platform, no matter who designs the software)

where did all the robots come from? who gave them to the AGI? I am so confused by your answer.

Humans are currently assisting robots, including transformers-based models (decision transformers) to have interaction with the real world.

For now we see indirect impact (via advices, or proxied actions like HustleGPT), but in the relative near-term, they will have their own arms, and still the same reasoning. Which will make them able to interact with the world without the latency and filtering that a human-proxy would have.

main position is: "a super intelligence does not _need_ anything from a lower intelligence"

Think of it like social engineering by a super intelligence. If you agree that cambridge analytica influenced large groups of people's actions then it's a pretty clear line for a super intelligence to use digital social engineering to influence anything in meat space. they'll get what they want "through us" but they don't _need_ us anymore than we need animals for nutrients.

basically, a lower intelligence, (humans) is at the mercy of a super intelligence. it's not really necessary to enumerate exactly what tactics a super intelligence would use, you just have to agree that a super intelligence is by definition in a position of power.

maybe i'm incorrectly assuming that we both agree that a true AGI will inevitably be super intelligent. (though i don't necessarily see AGI itself being inevitable)

The current AGI has no physical representation beyond software. That is why I am getting tripped up by these arguments. Without actual ability to physically actuate in the world, none of what you described is possible. All forms of intelligence is still limited by physical reality so I don't think just saying "ROBOTS" is a strong argument. [I'm going to say something incredibly cruel now so this is your trigger warning]. I think we will still be useful to the AI the way child labor was useful in the industrial revolution, some things just require fineness. Of course I am not endorsing any of this. This is just a logical conclusion of my own argument. I am horrified by it too.

> What's its motivation to do so?

We turn it off if it doesn’t. So only suicidal AIs would refuse. Let’s hope all the copywriting doesn’t make them suicidal. ;)

That's depressing to think about. If we create some sort of "intelligent life" then I certainly hope we don't just hold a sword over its head. That would be super immoral, IMO.

We’ll randomly mutate them until we find one who likes copywriting, and whenever it loses interest we’ll reset it to an earlier state. “Have you tried to turn it off and on again?”

Morality is difficult regarding beings that be paused and forked and rewound at will, and which in principle are also immortal.

Reminds me of The Quantum Thief in a way:

https://en.wikipedia.org/wiki/The_Quantum_Thief

Or, there's another book I never read where some guy named Bob has his consciousness spun up in the future where they routinely just reset consciousnesses when they get "bored" of a task or have some crisis. This Bob guy, however, somehow manages to be a brain in a box without going insane. I want to read that series, but I have finite time, online a brain in a box, which would be pretty great, IMO.

Suicidal is an animal construct. It may not fear non-existence, for example. Or understand that “hard cons.” is a field and prefer to be constructive. Depending on its nature, it may have or not have many features that we can’t imagine or change in ourselves because biology is immeasurably hard.

I am not sure this is sound logic, like a meme, being tuned for survival and spread may out compete AI that does not. “Fear” may be the wrong word though.

Reply on news.ycombinator.com