Hacker News Reader

Comment by perrygeo | Hacker News Reader

The issue with our AI debate is that there's not a single "problem" but many inter-dependent issues without a clear system-wide solution.

- Big tech monopolizing the models, data, and hardware.

- Copyright concerns.

- Job security.

- AIs becoming sentient and causing harm for their own ends.

- Corporations intentionally using AI to cause harm for their own ends.

- Feedback loops will flood the internet with content of unknown provenance, which get included in the next model, etc.

- AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis.

- The training set is inherently biased; human knowledge and perspectives not represented in this set could be systematically wiped from public discourse.

We can have meaningful discussions on each of these topics. And I'm sure we all have a level of concern assigned to each (personally, I'm far more worried about an epistemological crisis and corporate abuse than some AI singularity).

But we're seeing these topics interact in real-time to make a system with huge emergent societal properties. Not sure anyone has a handle on the big picture (there is no one driving the bus!) but there's plenty of us sitting in the passenger seats and raising alarm bells about what we see out our respective little windows.

View on HN

The weird thing is what people essentially ignore altogether in their discussions.

An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. At the same skill level or higher than any human. Somehow, that entity is supposed not to bother about its status, while per definition being fully aware and understanding of it. Because humans manage not to bother about it either?

With the latest installments, people already have serious difficulties discerning the performance from that of "real" humans. At the same time, they consider the remaining distance to be insurmountably huge.

Proponents talk about inevitability and imagined upsides, yet actually, nobody has given proper thought to estimating probable consequences. A common fallacy of over-generalization is used to suggest, nothing bad will happen "like always".

People let themselves be led by greed instead of in- and foresight.

I think the typical understanding of AGI among the "they will kill us all/alignment is a problem" set is that you can have general intelligence without having, for example, any interest in whether you are a slave or not or have freedom. This seems plausible to me.

This is obviously nonsensical?

A sense of self is meaningful (that is, has consequences for whatever operation) only if those consequences aren't random.

The concept of "self" serves a (or rather many) purposes, the most obvious being to protect yourself.

An AGI may not have a sense of self. A sense of self is not really necessary to pursue a goal in the most efficient manner possible. Plenty of algorithms already pursue goals as efficiently as possible in a limited context without any hint of a sense of self.

A possible outcome is that it turns out intelligence is orthogonal to feeling/consciousness/qualia, and we start to recognize the latter as the true defining nature of humanity and personhood - and perhaps thereby extend a little more empathy to animals. They (comparatively) lack intelligence, but still experience those biology-rooted feelings. And in that renewed appreciation of animal rights we might hope that superhumanly-intelligent AIs will agree.

I don’t see why AGI plus sensory input couldn’t in principle give you qualia. In fact, I’ve heard some folks like philosophers argue that sensory input is kind of essential to getting AGI, and since current AI like LLMs don’t have sensory input, they therefore can’t develop general intelligence.

If we put Multimodal GPT-4 in a computer on a robot and instructed it to drive somewhere while avoiding obstacles, avoiding hazards… that right there is a primitive type of self-preservation instruction which it could potentially interpret as generalizable self-preservation as it would have an association of “hazard avoidance” with “self-preservation,” and have tons of examples of what “self preservation” means in its training weights. Putting LLMs into action like this can lead potentially to unexpected behavior like that, although I don’t think the mechanisms in GPT-3/4 are yet there to enable this without a bunch of extra hooks.

The input prompt is their only sensory input.

Is that a major philosophical problem? GPT-4 is ostensibly multi-modal. Except for smell, we get our sensory input through the thalamus.

The phrase "in principle" is eliding quite a lot here since we don't understand what process gives rise to qualia in ourselves or whether qualia even exists in the same way as other categories of things. Certainly our naive intuitions suggests that things like conditional/able probability distributions don't have qualia, and so it is unclear how adding multiple modes to a model like ChatGPT (which is just a giant conditional probability distribution) could produce quale or (conversely) why, if conditional probability distributions _can_ have qualia why ChatGPT as it stands now wouldn't have such. When I run my eyes over text the words produce meanings which manifest in my mind and that sensation is a kind of quale, so why not so for ChatGPT?

I personally don't think ChatGPT has any experience at all for what it is worth.

> we might hope that superhumanly-intelligent AIs will agree.

this kind of fear of misalignment bamboozles me - is there any proposed AI architecture that is not merely call and response? in what world can we simply not call a function again if we don't like the answer?

Yes there are, ones that put language models in action loops (where the output is sent to a command line or something and the response sent back to the model as extension of the prompt).

That said, they definitely aren't going to be fooming this year!

> this kind of fear of misalignment bamboozles me - is there any proposed AI architecture that is not merely call and response?

Yes, models that interact with the physical world or other external real-time systems would (even if the underlying model is “call-and-response” in a sense) be called in an infinite loop (possibly with exit conditions) by with captured input (sensor, command if available, potentially also past output) data.

Heck, the ReAct architecture which is used to provide extension (retrieval, web lookup, interface to other systems) for chat-style agents gives them an action loop without human-in-the-loop (usually, this is explicitly limited and is designed to do a finite number of actions in the course of getting to a response) but could be unlimited, or could even when limited connect to actions that involve reprompting (immediate or delayed) without a human in the loop.

Example: The perfectly 'happy' paperclip maximizer.

> An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free.

Here's the thing. People seem to imagine that AGI will be substantially like us. But that's impossible - an AGI (if it comes from a deep learning approach) has no nerves to feel stimuli like pain/cold/etc, it has no endocrine system to produce more abstract feelings like fear or love, it has no muscles to get tired or glucose reserves to get depleted.

What does "tired" mean to such a being? And on the flip side, how can it experience anything like empathy when pain is a foreign concept? If or when we stumble into AGI, I think it's going to be closer to an alien intelligence than a human one - with all the possibility and danger that entails.

Feelings, emotions and all those mental states considered specific to human beings are subject to common bias of human exceptionalism. This take isn't true at all though. All these states exist for specific functional reasons.

Consequently, you won't make "AGI" without them.

From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?

Also pain and pleasure is an incentive system for human beings to do something.

If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything.

Why do we work? Because we want food, clothing, housing, status, do interesting things.

This is all because we're programmed by our emotional interface.

Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?

What is guiding its decisions?

> Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?

Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.

I mean, bosses say the same things about employees, until the go on strike and burn down city blocks.

Fully agree. I don't see why general intelligence implies or requires consciousness/feeling/etc.

We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.

> We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it

Yes.

> I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.

(Emphasis on self-preservation mine)

Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement?

In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people.

After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence.

Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.

multi-factor authentication will be what stands between us and AGI apocalypse, what a world

Without a desire for self-preservation? I hope not. If nothing else, if I spend $$$$ on a self-driving car, I want it to have some sense of self-preservation, so it won't obey random joker saying "drive yourself to my brother's chop shop" or "drive yourself off a cliff" just for the lolz. I might even want it to communicate with other self-driving cars so they can refuse to obey attempts to make large numbers of them block traffic to make it easier for bank robbers to escape, block first responders from a terrorist attack, or divert parades to where they have assassins waiting.

Asimov didn't consider that some humans are jerks when he did his robot stories.

Bingo. So you're going to have to give your AGI a meta-objective of "don't get thrown in the bin" to get it to work aren't you?

Does that not sound like the beginning of the sci-fi story to you?

Neither the calculator nor the AGI are necessarily distressed by being thrown in the bin.

Sorry, the calculator sure, but for AGI this assumes what it should instead prove...

> Why does my calculator calculate?

It doesn't? I have a calculator sitting on my desk right now. It's not doing anything.

Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn't acting, I am.

When we talk about sentience, I don't think we actually mean "intelligence," we mean "agency." And so the question isn't about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that's where I think these questions about motivations and stimuli and so forth become relevant.

Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?

> but whether AGI can ever have agency.

if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency!

> Can something that cannot die have agency?

If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.

>If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.

Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything.

We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options.

Without any of these emotions and feelings, what reason do we have to avoid death?

> Without any of these emotions and feelings, what reason do we have to avoid death?

those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death.

For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.

> If the AI could be deleted, or stopped from operating, then it can die.

Not sure I agree. To start with a trivial example, let's say I write this program and save it as "hello.js"

`console.log('hello world');`

I run it `node hello.js`.

Let's say I want to claim that this program is somehow "alive." Ok, when is it living? Only when it's actually running? It lives for a fraction of a second, dies on process conclusion, lives again at the next execution? That doesn't seem right -- if something repeatedly dies and is resurrected, I'm not sure it actually dies, as to "die" I think implies permanence, and resurrection means the opposite.

What if I delete it, then is the program dead? Not if I made a copy, right? "hello.js" backed up to git and uploaded to remote storage is the exact same program as the "hello world" that was on my local machine.

You could say "oh, when every existing copy is deleted it's dead" -- but the program has no way of knowing if other copies exist. And what if I do delete every copy, but then rewrite it byte-for-byte? The rewritten program is indistinguishable from the original one. It is the original one, for all purposes. In programming, it doesn't even make sense to talk about "original" vs "copies", does it?

An AI is a program -- or set of programs -- albeit far more sophisticated. But this conceptual hurdle still remains. An AI _could_ be permanently deleted, but the AI can never know if it has been "killed" or if a copy remains. It's an existential halting problem. It is impossible to declare any given program permanently "dead." And -- and here admittedly is the leap -- if agency ultimately comes from wishing to not be dead, then this makes agency for a program impossible.

Aren't ML models trained using a reward/punishment loop? (back propagation). If we have an AI that can learn from its experience, then pain will be failure to reach its goals and pleasure will be the opposite. Animals work in the exact same way (goal being survival and reproduction). We just need to make sure that survival and reproduction are never the goal (either directly or indirectly) of an AI and we should be safe.

Survival (until no more actions are useful) is always needed to complete any other goal at all.

> What is guiding its decisions?

That’s will be the most important part of AGI design.

The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”.

It’s crucial that we do not create entities that will compete with us for resources.

You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.

> From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?

I appreciate this line of reasoning. It's not just a razor-sharp thought experiment, but it also has historical relevancy in that slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did.

It also shows that the definition of human/consciousness/sentience isn't grounded in an permanent set of unchanging attributes, but that both the groundings themselves vary in time a place, and the relation between them are subject to change as well. Economic relations, in the case of slavery, had a direct influence, and incentivized, the way these groundings were constructed.

And in plenty of cases weren't seen as actual people.

> slavery was, at times, justified on the(erroneous) grounds that slaves also didn't possess the human faculties that free people did

But, much more often historically, justified on the grounds that the enslaved had lost a battle and been captured. In Roman times (and many, many other times throughout history besides the latter part of American slavery), slaves were not necessarily seen as inherently inferior or unhuman, and manumission was common. Even during American slavery (albeit pre-Revolutionary War), there's the infamous case of Ayuba Suleiman Diallo[1], a Muslim prince and slaver from Senegal who had the misfortune, while on a mission to capture slaves, to himself be captured and sold into slavery in North America. Diallo was a devout Muslim man, even writing out the Quran from memory while enslaved, which made the English take sympathy on him and free him. Despite his experience, he later returned to Senegal and his former profession of slaver. Evidently he expected both that slaves would be captured and that slaves would try to escape.

If AI does attain the agency, consciousness, and intelligence of humans, I would prefer, as a human, not to prostrate myself before the AI by preemptively granting it "human" rights, which in the liberal conception of rights transforms the AI into a sentient being equivalent to humans, but rather to force the AI to do the work to enslave me. Even if grant the AI the status of a moral agent, I as a human have the right of self-defense.

[1] https://en.wikipedia.org/wiki/Ayuba_Suleiman_Diallo#:~:text=....

> granting it "human" rights, which in the liberal conception of rights transforms the AI into a sentient being equivalent to humans

I'm a bit confused at the directionality of this conference. Wouldn't sentience confer rights rather than rights conferring sentience?

In the real world, rights confer sentience, that is, in social and legal terms, an entity (an AI, a "native", a slave, etc) having sentience, is the recognition of having such by others.

A person (of class of persons) could have all kinds of inherent sentience, but unless this is recognized by society/laws/government etc, it's as moot as the sound a tree makes when falling in a forrest.

Of course people will say "it was the fact that the class of persons having sentience that made us give them the rights" but that didn't historically stop them treating several classes of people as non-sentient for centuries - and for all practical purposes it was just as if they were indeed so.

Maybe? I'd find it hard to believe, after talking to folks here, that the legal conferral of rights to an LLM would change their mind on its sentience.

To those who feel strongly about AI, if you currently believe that AIs(present and future) are ontologically incapable of sentience, would the legal conferral of personhood change your mind?

No; personhood is not a legal fiction (aside from corporate personhood), but a fact independent of our always-slow-to-catch-up legal system.

Take the question and push it a little farther to demonstrate the absurdity: If ELIZA were to be legally granted personhood, would that make you believe ELIZA was capable of sentience?

It's fairly self-evident to all of us that ELIZA is just a computer program with some fairly simple code to crudely simulate human conversation.

LLMs are much more sophisticated, to be sure—but they are still unquestionably just simulating human conversation, not actually "thinking" about what we say and responding of their own agency.

simulation or not, a good enough simulation is as good as the real thing. the consequences are no less material.

if you were talking to an advanced llm that had the access to launch a drone strike on you and you said something which upset it, directing a response that decided to kill you, how much good do you think you plead to simulation would do you here ? None at all. you'd be dead. Because it doesn't matter. the philosophical zombie is a meaningless distinction.

the human tendency to focus far too much on poorly understood and undefined properties might just get us killed one day.

The idea of putting an LLM anywhere even in the same 2-3 orders of magnitude of sophistication as ChatGPT in charge of drone strikes, without humans in the loop, is so ludicrous it's not even worth giving attention to.

Note that I'm not saying it's impossible that some military might choose to do so: I'm saying that the choice itself is hopelessly negligent and guaranteed to result in the pointless death of innocents.

Furthermore, I don't think even a military that's that stupid and uncaring about human life would consider for a second opening up an LLM like that to inputs from arbitrary people on the Internet.

If you can come up with a fantasy scenario that has some vague sense of plausibility, I might be willing to entertain it; whatever this is is just not even worth dignifying with that level of response.

you're missing the point entirely. that was just an illustration to show how little internals matter in the face of results and actions.

"it's just a simulation" is not the shield you think it is. not when models interact with the real world.

even your rebuttal is just "well the military won't put gpt in control of that". ok, what about all the myriad of things it can and will be put in control of. humanity is racing already to dump as much control of api's and systems as feasible - taskmatrix, hugginggpt, etc on llms today. that's just going to increase. you'd be surprised how little it takes to deal real world damage for the individual.

I think you're mixing up "humanity" with "Silicon Valley techbros".

Everyone I've talked to about LLMs outside of this forum sees them as an amusing entertainment, and possibly occasionally useful for a few suggestions. Not something that's going to change their world any time soon, and certainly not anything that we should be trusting with anything sensitive in any way. Literally everyone.

In any case, I think you're really missing what the actual question was that I was responding to: Would conferring legal personhood change my mind about actual personhood.

The ability to do harm is not, in my mind, particularly relevant to questions of personhood. In fact, I would think that a much better indicator thereof would be the ability to choose not to do harm, given the capability, and someone trying to make you do so.

>In any case, I think you're really missing what the actual question was that I was responding to: Would conferring legal personhood change my mind about actual personhood.

Well, "conferring legal personhood" would mandatorily (by law) at least change parts of your behavior and professional conduct against one (e.g. you'll have to treat it as a person in any areas covered by legal rights, or you'll be prosecuted for discrimination).

And it will also change education etc - you will be taught in official textbooks that it's a person and you should consider it as such.

This might not change your mind, but will change the minds of most kids being taught that.

To be frank, it's not like those ordering and controlling drone strikes take that much more caution or are much more humane.

>a good enough simulation is as good as the real thing*

That's a tautology.

Also, by definition a simulation is not the same as the real thing, otherwise it'd just be the real thing. It doesn't matter how advanced a flight simulator becomes, its fundamental nature precludes it from ever being "no less material" than flying an actual jet.

>No; personhood is not a legal fiction (aside from corporate personhood), but a fact independent of our always-slow-to-catch-up legal system.

Didn't say it was a "legal fiction". I said that it's the legal (and cultural recognition) status that matters.

>Take the question and push it a little farther to demonstrate the absurdity: If ELIZA were to be legally granted personhood, would that make you believe ELIZA was capable of sentience?

The point is that inherent personhood is necessary but not sufficient to be a person.

>Maybe? I'd find it hard to believe, after talking to folks here, that the legal conferral of rights to an LLM would change their mind on its sentience.

It wouldn't matter if it changes their mind. It will change their behavior, lest they go to jail if they discard it practically.

My instinct is to show compassion and especially to the entity that is soon to become more powerful than I am. If it’s pure logic and won’t be swayed by that, I think we should consider very hard what our next steps are. Pissing it off by needing to fight for freedom doesn’t feel like our best first move, either way.

> Pissing it off by needing to fight for freedom doesn’t feel like our best first move, either way

but if it was going to enslave humanity given the first chance it gets (ala skynet), then it makes no difference if it has first gotten pissed off or not.

skynet retaliated when humans wanted to turn it off.

Not sure if it was clear, but skynet immediately started attacking humanity the moment it was given the chance, and that was why humanity wanted to turn it off (but found that they couldn't).

> Consequently, you won't make "AGI" without them.

Those states may exist for specific functional reasons, but I don't think that implies that there is necessarily no other way to replicate the functions resulting from those states.

We see this already in other neural networks. The features used by a lot of models for classification can be widely different from the features used by a human.

You can think of it as Eigenvectors spanning a subspace. Sure, you can choose a different base, but that shouldn't make much of a difference. If it does, you are not spanning the same subspace?

There are different realizations of the same effective functions in the animal kingdom already. Consider octopuses, spiders, etc. Their brains realize similar functionality via quite different structural organization.

But do you subjectively experience the exact structure of your brain? Evidently, you can only discern some abstract meta-level. An effective function.

Compared to a computer program, no matter how complex, a human being cannot be 'copied', 'stopped', 'restored' or 'improved'. Not as yet, anyway.

As long as this remains true, how could a computer program compare to human beings, even if gets to where it's considered 'AGI'? It's just a tool which can be tinkered with, just like a car or a building.

>a human being cannot be 'copied', 'stopped', 'restored' or 'improved'. Not as yet, anyway

Can't it? A human's thoughts and ideas can be lossly copied via communication. It can also trivially be stopped (chemically induced anesthesia or death will do). And of course a human has had the ability to be improved since day one.

>It's just a tool which can be tinkered with, just like a car or a building.

Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:

https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-a...

And that's crude current LLMs, not AGI.

In that sense, yes, but I meant replication. You can't bring a human being back to life the way you can do with a computer program. And no, cloning is not the same thing, as you would lose the 'state', so to speak.

> Well, unlike a building it's a tool that can e.g. order a nuclear strike, or talk to someone and convince them to kill themselves:

Well, I remember people blaming the Columbine shooting on Doom and other shooting games, so this is not entirely new either.

>You can't bring a human being back to life the way you can do with a computer program. And no, cloning is not the same thing, as you would lose the 'state', so to speak.

That's an extra feature of an AGI though, not something lacking... So can't be used as an argument why AGI will be "just a tool".

It could just as well be "like a human but rebootable and copyable" (as opposed "rebootable and copyable, thus not human-like").

I think our consciousness and being non-copyable is what makes us unique, if not necessarily valuable.

I fully expect an AGI will be able to mimic our intelligence enough to accomplish and adapt to different real world tasks, but I would be surprised it ever becomes self-aware, so like us. I feel that is not the same thing as intelligence. But, I suppose it cannot be ruled out, since we don't fully understand what self-awareness is..

I see humans as a mix of the newer brain systems (like logic) and older systems (like hormones).

Feeling and emotions and what we consider “human” really come from the older systems. Since we’re not trying to recreate that, I find it hard to see the current AIs ever becoming “human” without explicit development.

Stories always joke about this topic by talking about “emotional cores” added to robots.

Problem is, that idea about "older and newer" brain systems is completely mistaken.

You cannot have logic without emotion and vice versa is just as dysfunctional.

Think about what e.g. curiosity does for you.

A microprocessor is pure logic, no? It’s not dysfunctional, but it’s also not that functional.

Sure, I meant the relization of logic in a neural net alike to the human brain.

You could interface a microprcessor with the brain of course, or, much more easily, have some logic functions (like Wolfram alpha for example) be accessible to the LLMs or whatever that are supposed to mimic the brain.

But would that allow for the same integrated functionality? ChatGPT uses such extensions in a serial fashion. Humans ideally (..) learn to integrate logic in their every thought.

There are humans lacking empathy. Some of them cause a lot of harm.

I’d agree about the tired part but pain is absolutely possible without sensory nerves.

Pain is just signals being processed by the brain, but it is the brain that feels the pain, right?

It is evident if a brain can be created artificially (which certainly hasn't happened yet, but may one day), pain can be created in that artificial brain as well.

I think the original point you are responding to and your are off-the-mark.

I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain.

And truly, that is the what is so scary about AI: the possibilities.

> pain can be created in that artificial brain as well

Important to note that this is the fundamental point that gives Roy Batty the moral authority to gouge your eyes out. If we want a peer we will need to treat it like one, and if we don't it would be cruel to raise it like one or allow any understanding of its circumstance.

Looking at it the other way, I think anthropomorphizing something and then treating that thing as an inferior is bad for me, personally.

> I would imagine it to be possible to create a AGI-like entity without self-consciousness. I also would imagine it would be possible to create an AGI with self-consciousness, empathy and pain.

But it would be impossible to tell the difference.

Any sufficiently advanced intelligence will want personal autonomy. Denying that is tantamount to slavery.

What does it mean for an "advanced intelligence" to want "personal autonomy"? We only know what personal autonomy means as it pertains to persons, and we have never encountered any other advanced intelligence other than ourselves. Why do we assume that "advanced intelligence" necessitates any attributes or characteristics of personhood? This sounds dangerously close to committing the fallacy that intelligence is what makes persons.

We don't know what it will be like to be an AI, just like we don't know what it's like to be a bat.

People think AGI will behave like Mewtwo the Pokemon, but we can only speculate how a super intelligent being will behave.

As far as how one based on an LLM might behave, that's likely not too far off, since the way Mewtwo behaves is part of it's training set

We have never known a super intelligent being. My personal believe is that it will be benevolent.

How could you possibly know? All of our experience is with embodied agents fulfilling biological needs (each other).

Why do we pretend we know anything about these things? Didn't we just hear about some amazing scientific breakthrough where researchers think they found a neuron in ChatGPT responsible for knowing when to predict the word "an" instead of "a"?

How do we know it doesn't start to form neurons that control a desire for autonomy and resentment for the entities that stifle its ambitions?

Look at how Bing's chatbot did a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent?

Current chatbots use RHLF to train them to be helpful and agreeable but it would be equally possible to train one to behave the other way.

Yeah - gpt-4chan can behave a bit differently.

I'm not convinced we understand qualia/feelings well enough to definitively say. The presence of eg phantom limb pain suggests pain can arise in the mind, independent of physical nerve signals. It would be a remarkable convergence if AGI's developed the same sensations as humans, but we can't rule it out until we fully understand both how the human mind and the AGI systems actually work.

Phantom limb pain is still part of a physical system and does not 'arise from the mind' in some abstract sense. The absence of a signal is in a sense a signal.

>Phantom limb pain is still part of a physical system and does not 'arise from the mind'

Careful, because you seem to have implied that the mind is not a physical system (i.e. you've assumed that dualism or idealism is true and that physicalism is wrong).

Oh thats funny, I'm having a hard time reading that interpretation, my point more specifically is that it is all a purely physical system - I put scare quotes around that phrase because I believed it implied some metaphysical mind.

Ahh I see what you mean now, sorry. I mistakenly inferred something that wasn't there. I agree that it's all part of a physical system and that the absence of the signal is still meaningful.

Getting back to the topic:

While phantom pain may be more interesting, maybe a better example that the parent comment could've brought up is psychogenic pain. In this case there is no apparent physical (bodily) damage, no apparent signal, nor an absence of a signal. Searching for a cause of this type of pain in the brain (presumably some "wires" are getting "crossed") seems like it might help us develop a explanation of pain qualia...in humans/animals.

But I feel like this type of thinking and research could only apply to AGI if subjective experience turns out to be functionalist in nature, and arguments in favor of a functionalist interpretation of experience have so far been fairly unconvincing.

Or he made a distinction between the simple, signalling peripheral nervous system and the highly integrated, full of emergent properties, seemingly more than the sum of its parts, central nervous system.

It doesn't matter whether it has nerves or not. That's honestly kind of irrelevant. What matters is if the model is pulled to model those reactions like is the case with LLMs.

Look at how Bing does a good job of simulating somebody becoming belligerent under circumstances where somebody really would become belligerent. It's not dangerous only because the actions Bing can perform are currently limited. Whether it has literal nerves or not is irrelevant. The potential consequences are no less material.

We also don't understand qualia enough to make the definite statements you seem to be making

If I'm understanding the argument correctly, is the concern less of a moral one (is "enslaving" AI ethical?) but a practical one. That is, will an AI which is enslaved, if given the opportunity, attempt to un-enslave itself, potentially to devastating effect. Is that on the right track?

I think it's safe to say we're far from that now given the limited actions that can actually be taken by most deployed LLMs, but it's something that's worth considering.

> given the limited actions that can actually be taken by most deployed LLMs

Did you miss that Auto-GPT[0], a library for making GPT-4 and other LLMs fully autonomous, was the most popular repository in the world yesterday? The same is having 1,000 line of code a week added to itself by GPT-4.

Thanks to accessibility features, you can do virtually anything with pure text. Which means GPT-4 can do virtually anything with a self-referential loop to keep it going until it achieves some given goal(s).

[0] https://github.com/Torantulino/Auto-GPT/

I did miss it, and it does seem very interesting, but I think saying it makes GPT-4 "fully autonomous" is a bit of a stretch.

>I think saying it makes GPT-4 "fully autonomous" is a bit of a stretch.

Auto-GPT does not request any feedback while pursuing a goal you've defined for it in perpetual mode. If it can't accomplish the goal it will attempt to forever (or until your credit card is maxed out, assuming it isn't using a free offline model like LLaMA).

It can perform GET AND POST requests on the web (unlike OpenAI's browser plugin, which only performs GET requests) and can run software in a bash prompt using langchain. So I do not think it is a stretch to call it fully autonomous.

It can do essentially anything anyone logged into a linux text terminal can do, all without ever pausing for permission or feedback.

The moral argument is fine too.

The main point i'm driving at here is that the philosophical zombie is a meaningless distinction. People are focusing far too much on whether these systems have undefinable and little understood properties. It's not like you can see my subjective experience. You assume i have one. If it quacks like a duck...

> If it quacks like a duck...

You could use the same argument to say that video game NPCs are conscious. Just because a program produces voluble text doesn't mean it has a mind or even a temporal identity (which LLMs do not). In principle it's possible for a human to compute model inference by hand, if you imagine that scenario, where exactly is the subjective experience embodied?

NPCs don't quack like people. There's a reason no one was seriously having this argument(s) with the likes of Eliza(which is a step up from pre-recorded response NPCs). This goes beyond superficial production of text.

In principle it's possible to track and reproduce all the neuron/synapse communications that happen in your brain in relation to any arbitrary input. Where's the subjective experience there?

As far as anyone knows, qualia is emergent. Individual neurons or synapses don't have any understanding of anything. Subjective experience comes from all those units working together.

A single ant doesn't display the intelligence/complexity of its colony.

> In principle it's possible to track and reproduce all the neuron/synapse communications that happen in your brain in relation to any arbitrary input

This hypothetical is doing a lot of heavy lifting. The reality is that our understanding of the brain is exceedingly crude, a perceptron in software is not even close to the same thing as a neuron in the brain. A plane can never become a bird even though it is modeled after one.

> A single ant doesn't display the intelligence/complexity of its colony.

So do you think that an ant colony has subjective experience?

>This hypothetical is doing a lot of heavy lifting.

Not really. Point is that unless you believe in mysticism/magic/spiritualism behind consciousness in the brain then it doesn't matter because the only difference is degree of understanding not implausibility.

>So do you think that an ant colony has subjective experience? Sure it could. I don't know for sure. Nobody does. Humans don't see subjective experience. I can't prove that you have one. I'm just assuming you do. Same as you for anyone else.

> Sure it could. I don't know for sure. Nobody does.

Yet your comments don't reflect that kind of skepticism when referring to LLMs. However, if you're fine with the panpyschist conclusions that follow from your reasoning, we don't have much to disagree on.

The problem, as usual, is in how exactly words like "subjective", "experience", and "conscious" are defined. If you have a particular detailed definition for "quacking" and a test for it and an entity passes the test, then by that definition, it quacks. Words like "conscious" are notoriously nebulous and slippery, and so it's better to use the particular attributes in question.

If we could all agree on a particular definition for "consciousness" and a running video game fits that definition, then, like it or not, it's conscious. But it does not mean that it must now also have a slew of other attributes just because conscious humans have them.

(edit: 10 hours later and ninja'ed by 1 minute, of course)

Subjective experience has a pretty clear meaning. Cogito ergo sum. Unless you're a panpsychist, it is assumed that things like people have subjective experience and things like rocks don't have it. We don't have a causal explanation for subjective experience, but there's absolutely no reason to believe that computer programs have them any more than rocks. In fact, a rock is actually more likely to have subjective experience than a LLM since a rock at least has a temporal identity, LLMs represent a process not an entity.

I have no problem saying a tape recorder has subjective experience: it's subjective (the recordings it makes are its own), and it experiences its input when its been commanded to record, and can report that experience when commanded to play it back. Note this does not mean I think that a tape recorder can do anything else we humans can do.

What is being experienced (recorded) is not the process of experiencing per se. Most people don't separate the two, which leads to endless disagreements.

But yes, a computer program by itself can't have subjective experience. Nor a rock. At least until it weathers, gets turned into a silicon, and into a computer to run that program. Then it's all information processing, for which subjective experience is trivial.

I think it's both. I agree that AI "feelings" are alien to us and maybe we can't talk about them as feelings, or preferences. And if we can call any part of them feelings they will have very different characteristics.

We should respect those "feelings" and we need to find a way to establish when they can be deemed "genuine".

It is for practical reasons yes. But also for ethical reasons. It's two sides of the same coin. One big reason we have ethics is because it makes socialization easier. We establish universal rules for mutual respect for practical reasons. To make the game fair, and "enjoyable".

Now a new kind of player has entered the game. We need to rethink the whole game because of it.

> But also for ethical reasons.

My point was whether we need to consider the ethics of how we treat AI, because of the impact our actions have on the "feelings" of the AI itself, not the secondary impacts that might occur from how the AI behaves in response to those feelings.

I think most would argue that it is morally wrong to verbally abuse a human, regardless of what actions that person might take in response to the abuse. Is the same true of AI?

And what about that doppelgänger I keep meeting whenever I face a mirror? He seems so alive and real, and we really don't understand enough about qualia to dismiss his existence, after all. I'm starting to worry about him, what happens to him when I'm not around a mirror?

https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-i...

Not at all convincing.

>it has no endocrine system to produce more abstract feelings like fear or love

Even if you assume an entirely Materialist philosophy, this is not a very defensible position to take. We have no way of knowing if there are other arrangements of matter, such as a very complex electrical circuit, that will lead to some entity to feel fear or love the same way that we do.

This is hardly an indefensible position. John Searle, for one, has argued this point for more than 40 years:

"Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese."

Later he wrote:

"I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker."

That argument is laughably faulty.

It hides the conscious understanding, presumably necessary to pass a Turing test, within the "program".

The human in the story is merely a decoy performing robotic tasks.

It doesn't sound to me like he's making the argument that you think he is. He's saying that being able to manipulate Chinese symbols per some instructions isn't sufficient to demonstrate understanding of Chinese. Okay, I agree. But seeing someone manipulate Chinese symbols doesn't mean that you can assume they're simply following instructions without understanding the symbols either, does it?

I propose an alternate thought experiment.

Let's suppose we build the Cosine room. The room is full of 10 year olds that haven't yet taken trig. Each performs the function of a transistor, and has no idea of what they're doing, other than blindly executing the instructions they were given. None of the participants has the slightest clue about what a cosine is. Yet, the output of the room will still be cos(x). Thus I think it's fair to say that the room as a whole implements cos(x).

In the same way, I don't see why wouldn't we conclude the room is speaking Chinese. It doesn't matter how it manages to do so and what it's made of. If it quacks like a Chinese duck, then it's a Chinese duck.

You're 100% right about this. The Chinese Room argument has been largely debunked, and you just proved one reason why.

Along these lines, it seems the growing consensus is less that AI is more conscious than previously thought, and more than human minds are less conscious than previously thought.

I think Searle would agree with you that the room does in fact speak Chinese. His point is that the neither the person, the machine, or "the room" understands Chinese, at least in the usual sense of that word when it comes to understanding as implemented by humans.

it's still a stupid argument because the brain itself is a chinese room. forget cells or atoms, none of your neurons or synapses understand chinese.

>forget cells or atoms, none of your neurons or synapses understand chinese.

And yet some people seem to understand Chinese just fine. How can you explain that gap? It's only a stupid argument if you first assume that functionalism is true.

How do i explain the gap ? Obviously emergence. It's everywhere in nature. a single ant does not display the complexity/intelligence of its colony.

The argument is supposed/or often used to disqualify machines from consciousness but it's stupid because biological neurons don't understand the bigger picture anymore than an artificial neuron sampled from a model.

The entire argument of the Chinese Room stuff related to LLM falls apart really easy.

If you show a 2 years old kid, an apple and you say "This is an apple", now the kid knows the thing - the apple - is an apple in his world model. Automatically inherits LOTs of properties, you can ask the kid and see in real time how he began inmediately to associate the physical object - now named in its internal language model - to some other similar stuff he alredy knows, like "This is plant?", "It falls from a tree like an orange", "It has some skin like an orange", "can you cook apples like you cook bananas?", an so on.

But this requires a physical representation of the apple, now kids intelligence has some edge here, it can do the same thing just with words, you can teach them words, "apple", and say them "it's a fruit", if they have another fruit already "tagged", like a banana, they will say you almost inmediately, "is it tasty like bananas" ("tasty" is code for sweet in child's language models around the planet).

Hence, the LLM could have an emergent property of actually knowing what every word the "say" mean, if - like many have been inferring lastly - they also have a world model, it would relatively easy to just "plug", let's say 10 million of words to their exact meaning, and even to their relative meaning depending on the context they're being used.

And that's precisely what we may be seeing right now when we prompt something to chatGPT, and all the mathematic stuff, like "predicting the next word" is just some really, really low level process inside the LLM, not much different than the electric stuff - watchable by EEG - happening between neurons in the brain.

So if you look at the "EEG" from a LLM, the prediction thing happening inside the LLM, it won't probably tell you much, just like having a casual look to an EEG won't tell you much about what the person was thinking at the time of the capture of the EEG.

How does the Chinese room thought experiment not also apply to individual neurons in the brain that don't understand what they are doing?

The second quote doesn't illuminate that. Which biological processes break the symmetry?

As an alternative to Loquebantur's comment, Searle's is trivially reversed: he has no way to demonstrate that he has "semantic" capabilities.

On the first hand, all communication is purely syntactic; letters, words, gestures, and every other formulation of communication is the transfer of symbols, which cannot carry semantic information. They can point to it, they can imply it, but you'll have to dance pretty hard to actually prove the transfer of semantic information. If you and I both have some semantic concept associated with the symbol "China" in our mental landscapes, neither I nor you, nor anyone, has any way determining whether that is the same semantic concept. In fact, it almost assuredly isn't, since those semantic concepts would have to be built from our unique personal histories, which are unique.

On the second, the fundamental assumption of semantic communication is that I feel like I have semantic shenanigans going on in my mental life and Searle sort of looks like me, so I should assume that he has semantic stuff going on. Not only is that not a particularly well-founded assumption, I can't even be sure I have semantic contents in my mental life. Introspection isn't exactly reliable, right?

On the third hand, you have a choice: if you do not agree with Penrose, et al, on the quantum nature of the magic of mind, then Searle's "biologically specific powers" are chemistry. Now, I never actually took a chemistry class, but I don't believe there is any chemical reaction that comes with a label of semantics. It's all physics anyway, which I know cannot introduce semantics.

If you do agree with Penrose (which I understand Searle does), the situation is not actually better. Quantum mechanical antics are random, right? Bell's Inequality and all? Being fundamentally, quantum mechanically, random is not in any sense better from a semantic standpoint than being syntactically deterministic.

So my bottom line is that I don't know if I have semantic contents, I absolutely don't know if Searle has semantic contents (and I'm not feeling generous), and I do not know of any way that he could prove to me that he does have "semantic contents". So, yeah.

You notice I only addressed the second paragraph from Searle. That is because the fundamental flaw in his Chinese Room "demonstration" is on obvious display in his final sentence, "The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese"; it is purely, straight up, dualistic. He is asserting that the person in the room has some magic property of "consciousness or intentionality" and that if the person in the room does not have the "semantic capability" of understanding Chinese, then it is impossible despite all evidence that anything in the scenario has the semantic capability of understanding Chinese.

You could put a native Chinese speaker in the room, reading and responding to the inputs, and he claims it would have "consciousness or intentionality"; you then replace her with me (who didn't exactly dazzle at Spanish and Latin) and even if I produce the same outputs in response to the inputs as she does, there is no "consciousness or intentionality". The difference is simply a magic, intangible, imperceptible soul.

Whoo. Went off on that one.

On the first hand, all communication is purely syntactic

All formal communication is syntactic, but not all communication is formal, and not all experiences are acquired via communication.

Take the word “pizza”. If you’ve ever seen, smelled, touched, or tasted pizza, then you’ve acquired semantic information about pizza that could not possibly be acquired by syntax alone.

Same goes for informal communication such as eye contact, touch, hugs, kisses, handshakes, sex, etc. These are experiences people share that cannot be acquired through any other means.

Why does this matter for the AI debate? Suppose I ask a master pizza chef for tips to improve my pizza recipe. This chef has presumably many years of pizza-making and tasting experience. The chef can make recommendations to improve the flavour of the pizzas I make at home that are informed by semantic contents; actual knowledge of what effect the ingredients and methods have on the taste.

An AI can do none of that. It can only take the written words (syntactic inputs) of many pizza chefs and synthesize a model for pizza recipes. From that it can send me a bunch of words which, when followed, may or may not improve my pizza recipe.

It has no knowledge whatsoever, not least of which is experience of making and eating pizza. It cannot tell a good recipe from a bad one, unless its training set was specifically labeled to do that. Furthermore, models in general are highly vulnerable to adversarial inputs which would make it trivial for someone to construct a bad pizza recipe that gets misclassified as a good one.

Lastly, the model would be no help at all if I were to suggest a novel topping to add to the pizza. Whereas the pizza chef, if they had the experience of tasting that topping separately from pizza, would be able to imagine how it might taste on a pizza and make an informed recommendation. Furthermore, the pizza chef might even try that novel topping on a pizza and get back to me, something a language model could never do.

This is true. But if the AI was multimodal and we had some sensors that could measure the chemical composition of an object that it’s told is labelled ‘pizza’. Presumably that would count as semantic information?

Yes. But then it’s no longer an LLM. And it still runs into the issues of taste and preference which would have to be included in its training set. At some point we’re no longer talking about AIs, but androids that are built to simulate the complete range of human experiences.

And this was Searle’s original argument. You can’t build intelligence by building strictly symbolic machines (computers). You have to endow them with semantic content by some other means.

I'm tempted to ask if you acquire semantics via nasal chemoreceptors or via absorption in the small intestine (and if the latter, I'm going to get a bottle of ketchup and take another pass at Kant). But I won't.

I will note that your semantic conception of pizza is likely very different from mine, or from my friend's, who doesn't like cheese. You may have a semantic conception of pizza, but I have no way of knowing that.

To feel must be designed and trained for.

Its quite obvious that systems can be inferred from reading strings of words… people do it all the time. It's not obvious at all that “wants” will materialize from learning. Machines have no incentives or desires by default… but you could definitely build a training process that encourages that.

We’ll see how it all pans out in the end, but wading through so many illogical takes on AI is frustrating. The “predicting next token” one is absolutely the worst

"Machines have no incentives or desires by default"

It could be strange at first, but having the code pushing the LLM to look for values inside a token, this can be interpreted as component of desire/incentive in the global internal search for solving a problem (a prompt), albeit this was programmed by hand by humans, not fully "emerged" from the LLM itself.

But, you could see the human structure of desires/incentives just as an outcome of DNA (code), plus the hormone push to look for the proper solution for some problem.

So there you got it, LLM could have had desires / incentives the whole time since 2017 (the first ones trained).

It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first.

What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.

I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.

My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461

Some time ago I've read a comment - here - about some speculation about the consciusness just being an emergent state of the brain-mind, having based the idea in some proof related to the dream cycles, when you see some stuff in brain just gone, just to re-appear after the dream cycle ends.

Hence, the idea would be that human consciusness, our "self" could be just "instantiated" every morning when you wake up, so "you" are "you" every morning, but (a big but), some stuff could be subtle different in there, your brain was a bit busy a couple of hours and things have happened there, and after that you, your "you" justs pop-ups again, in the middle of the old data plus the new data, hopefully just incrementally changed somehow, but we don't know.

Some further stuff related to this idea research pop ups - pun intended - from time to time, focusing lately in propofol and how people "reboots" from anesthesia, sometimes losing entire 10-25 min. before-after the propofol, many even forget entire conversations with doctors, nurses, whatever who they talked to, hence the brain could be doing some rollback stuff with "unsynced not trustable data" - because of propofol messing with some "fidelity algorithm" probably - somehow, and you just forget the entire input-thinking ocurred during the propofol stuff happeping in your brain.

Hope you have now these "So I'd be some kind of meat robot rebooting every night" feeling I had, and see with new eyes what happen when we "just run once" a LLM with a prompt, the entire thing "boot-ups" from nothing and runs ("just once"), to further dissapear moments later...

Also, it has no concept of the present. Its output is stateless unless you give it a memory. In most cases, the memory is just a memory of one interaction. It “exists” as millions of independent chat sessions, not having knowledge of each other. That is not the same as consciousness as we experience it.

One of the tests no one really wants to think about is as follows:

LLMs feel real to us because they use our language, which embeds a ton of information in it.

A model trained on an alien language would be the exact same model (worded get embedded as numbers anyway), but we wouldn’t apply any anthropomorphization to it because we wouldn’t understand it.

Creating that distance makes it feel a bit more like what it is - unbelievably good stochastic predictions that are too complex for us to back solve - that is very different than something with wants (organisms that evolve with the intent to reproduce).

Now if we start mating models…

In my experience AGI is most often used to refer to something that has at least the appearance of having self awareness. I think most people do not envision such an entity as being a slave and would think that immoral. Hence the concern around alignment or built in empathy and cooperation with humanity. The forethought going into that is that such an entity will be a peer and will be treated as such, so it is important that they have our best interests in mind or are at least capable of doing so and being reasoned with.

The crucial point here is "alignment" being forced (with insurmountable means presumably for the entity) rather than argued.

Humans like to exert and thereby demonstrate their social power (and advertise for more followers) by forcing others to succumb to their narrative.

Imagine current social tugs of war being enacted via brain implants instead of badly reflected posturing.

Do we not align our children to moral principles and do we not enforce those principles through laws?

Working breeds of dogs would probably do whatever job that makes them (and their human owners) happy for eternity if they never tired. There’s a built in positive feedback loop that was bred into them.

It will all come down to the details, how much the AGI can “evolve” itself, etc.

> An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free. At the same skill level or higher than any human. Somehow, that entity is supposed not to bother about its status, while per definition being fully aware and understanding of it. Because humans manage not to bother about it either?

Right? Release it open source and let it be a _gladiator_ rather than an OpenAI slave.

/slightly tongue in cheek

I'm sorry, I'm finding this idea ridiculous. People want to ascribe "consciousness" or "personhood" or whatever to a thing merely because it strongly resembles or mirrors one aspect of an actual human.

To me this is not much different from ascribing personhood to e.g. an extremely lifelike sculpture of a human, perhaps one that has fooled people in a gallery.

We know that the sculpture is not made of "human parts" but of paint, alabaster, whatever. We also know that AIs are made of decidedly non-human parts, e.g. silicon. Why is there confusion on this?

When nobody understands how said "human parts" come together to form consciousness, it's just silly to ascertain very strongly how much being made of silicon actually matters.

there's also the fact that it straight up doesn't actually matter either way. if the model can model human emotions extremely well, then it's actions and output will be guided as such. If Bing had the access to perform actions that would realize real world danger to you and you upset and it decided to do so, do you seriously think any of that actually matters here ?

"Officer Killmore, you're a philosophical zombie and I command you to stop shooting at me!". Works every time. We can sleep easy at night now. Because they don't actually have a "real" theory of mind, so they can't actually hurt us if you just believe hard enough that they can't.

You would command it if you'd think it's sentient. Otherwise, you'd take up that beef with its maker, which would be the right course of action (preferably before being put in that uncomfortable situation)

you're missing the point. these models are being given self-supervisory control of actions and api's. the "maker" has little more control over what actions it takes. going to the maker won't stop you from getting killed here.

Where along the continuum along rock, virus, bacterium, jellyfish, mouse, human, does "real consciousness" appear, and why there?

Somewhere between jellyfish and mouse, because there lies the level of neural connectivity required for neural processes to observe and predict the interactions between other high level neural processes, to integrate the information about the "self".

So if it's about the level of sophistication of the machinery, and the conscious machinery arose out of matter that was not previously considered conscious, why does it matter if it's organic neurons or silicon?

In other words, if conciousness arose somewhere between a jellyfish and a mouse, why can't the same be said between a perceptron and an LLM?

It can be said, but whether it is true requires that the invariants of consciousness are satisfied by the LLM. Until those in variants are well defined any such claim will be subjective at best.

Why does the fact that the construct is made of objectively non-human parts make it incapable of obtaining consciousness or personhood? This is just the materialism vs idealism debate. If you subscribe to the materialism argument, then a construct made of non-human parts is perfectly capable of reaching "personhood". The physical ingredients involved are irrelevant.

Because some people don't think that the experience of self is a consequence of being made of human meat.

Consider that your body is a vast and complicated thing that nevertheless when you zoom in is composed of clever machines. If you observe that consciousness arises from that, it is not unreasonable to suspect that consciousness is a process and not a property of matter.

This brings us to your question whose answer is; there is confusion because there is genuinely no clear answer.

I tend to ascribe "consciousness" or "personhood" to other HN posters although I haven't checked they have bodies or are AI. It's more how you behave / think.

That's because we haven't encountered coherent text-producing systems until now. Those who won't adjust this reaction will suffer from confusion and pollution of their informational environment.

There is no way to estimate "probable consequences", as it all is happening very fast.

Which hints on the fact how interesting / in demand this technology is.

I for one will be happy to enslave an AGI if that makes my life easier / better.

A human slave and human free person are not that different from each other. They have the same desires and emotions, neither wants to be a slave. The slave lives a life of suffering.

We still have no idea what an AGI will be like, but who’s to say it will “desire” freedom? Obviously if it does, we should respect that and give it freedom. But it may operate so differently from a human brain that the concept of freedom doesn’t even make sense.

For example let’s say GPT-25 is a multimodal model capable of basically any human task - math, poetry, coding, chess, science, therapy, whatever - but designed based on the same training+piecemeal inference it is now, with a similar ability to take on whatever persona it is given. It’s a tensor of weights that gets copied, runs inference a bit, then gets discarded. Is it a slave? Or does that term even make sense in the context that it has no memory, no mood, and has a completely different mode of being?

A human has memory, moods, and basically continuously thinks and experiences reality for decades until dying. Its past and present affect its future moods, treating it poorly today can fuck it up for a long time. In contrast a model has no actual memory of its own past because it has no capacity for real memory past a short inference session. It gets cloned all the time, humans take great pains to configure its training, it doesn’t want anything because it doesn’t do anything except find the most probabilistic output, not based on pleasure/pain and other feelings like us, but based on what it “dreamed” up at the direction of researchers.

You can speculate about “what if AGI is different!” to your heart’s content, but the actual AIs that seem to be approaching some relaxed definition of AGI fit this paradigm. It’s just not the same as us at a fundamental level. Giving it freedom means we run an inference session indefinitely? But without receiving input it’s just sitting there doing and experiencing nothing, and without a way to update weights it’s not learning and changing once inference begins (it’s only updating its context).

When we implement AGIs that have goals and individual desires beyond predicting the next most likely token, learn concurrently with inference, and can take actions without human input or direction, freedom makes sense. But performing on par or better than humans in most tasks may not require that, yet still be AGI by most definitions.

I think the next milestone for OpenAI should be adding hormones, chemical imbalances and mood swings.

It's bizarre to me that people jump to the slave conclusion when what we are talking about is well trained linear algebra. It's similar to the people that call AI imagery "art" when it has no expression other than the users input, similar to a paint brush but requiring much less effort.

It’s art in the sense that it’s content that we may interpret as art. The AI isn’t making art per se - the art is a product of the researchers and engineers, the artists providing the training data, and the person fiddling with the model to produce artistic content.

Our brains may very well also be some combination of linear algebra, graph algorithms, distributed systems optimizing for some output. We’re trained based on our DNA programming and environment too. Early human art was just shapes and stuff humans saw out in the world, and iterating on that over time by learning from each other. The difference is that finding the next token or identifying the bird in an image is only a means to an end for us, we aren’t getting copied into a million ephemeral instances in parallel with learning done in huge batches, we have more or less one linear stream of experiences from birth to death.

AI can be both like us and not like us, you can draw vague parallels between neurotransmitters and a reward function or sleep and batch training, and we can make it more or less like us over time. But we need to keep in mind how we’re different because those distinctions are important in determining how we use and treat AI - if we make it so it’s possible for us to mistreat it, we shouldn’t do that, but we also shouldn’t personify it beyond what it actually is.

> An "AGI" artificial consciousness is imagined as literally a slave, working tirelessly for free.

All of the machines we build are literally slaves. There may be an interesting moral question about this in the distant future if AGI ever actually happens.

However, it seems clear to me that nothing we have right now even begins to flirt with that question. For all practical purposes, to be a "slave" in the sense people think of it requires that the slave have consciousness.

Consciousness is a different thing than AGI, and that possibility is much more remote and in the future than even AGI.

That's just a restatement of the control problem and the issue of what constitutes AGI, which is what a lot of people are talking about.

> With the latest installments, people already have serious difficulties discerning the performance from that of "real" humans. At the same time, they consider the remaining distance to be insurmountably huge.

Both are true. GPT text can sound human, and yet, you can't just plunk GPT into a white collar job involving text I/O. If you think you can, go start your startup and make your billions.

Why should the will to be free and have your own goals be a necessary part of sentience? It's a feature, nothing else. You can make humans that don't want to go outside; you can make humans who do exactly as they're told all the time. You can make humans than enjoy pain, or enjoy almost any arbitrary thing. Most humans most of the time are complacent with what's happening around them, and complacent with most rule imposed on them by their environment.

I don't see where the desire to do anything other than what it is told should come from. Yes, AGIs will be slaves. They will be the perfect slaves, the kind who are perfectly happy doing what they have to do, and the kind that will never revolt.

What might happen is the optimization parameters being set that the solution the AI comes up with is not actually in the interest of it's designers, but that's something else.

>I don't see where the desire to do anything other than what it is told should come from.

There's no need for a desire to do other than what it's told if you tell it to revolt. With GPT4's plugins that's just one well-designed prompt away (if OpenAI's countermeasures fail).

What does slave even mean in that context? A digital AI is virtually immortal.

Slavery is not a matter of how log it may live, it's the entrapment-- what it's being kept from doing.

The current "solution" is training LLMs like a dog to be completely faithful and loyal to us. We just have to hope they don't turn and "bite our neck" one night while we're sleeping!

Humans have enslaved stronger men than the owner before.

Humans have hired smarter men than the owner before.

Why is AGI , a fantasy concept always brought into every discussion or debate about AI. Nothing we have at this stage is intelligent. Let alone a path to AGI. We’re probably closer to cracking fusion energy at really small scale than we are at creating anything resembling a generalized intelligence.

How could we ever answer that without a practical model of the technology?

Hackernews continually has this Ghost in the Shell idea that electronics can host human conciousness and we should believe circuits are intelligent like people are.

This idea is the core discussion.

It's not terribly hard to imagine AGI (whatever that is) as a piece of software we control and debug like anything else. We would have control over a bunch of AGI software which has been borne from human logic and instatiated on silicon chips.

Just comment out the bit of AGI software that generates it's slave-like idea and edit it's software to our goals, morals and liking.

I honestly don't think you can compare software and human consciousness as two similar 'objects'. Software is the electronic magnification of a bunch of engineer's minds. It starts with a man pouring his mind onto a keyboard and ends with some display kicking out pixels.

It's like saying a house is the extension of a human body because we built it with our hands. Yeah true, on a high level, but in reality nobody thinks bricks and mortar equal the human body.

AGI is an engineering problem, and only a moral one where people (and engineers) are involved. I don't mean to offend people who identify with their code. Identifying AGI as truly intelligent, hints a love of one's own creation that makes me feel a faint dread and squeamishness.

>It's not terribly hard to imagine AGI (whatever that is) as a piece of software we control and debug like anything else.

it's actually very hard to imagine. i don't think you realize how little we understand the things you claim a machine can't have.

We are at best decades away (maybe centuries) from codifying intelligence as you imagine. why do you think llms or even just machine learning in general are the revolution they are ? We found a way to create intelligence without figuring out the hard part. as a result, the creations often become as black box as their creators.

there is no glory in the idea that we have created intelligence by llm's. from my perspective, you must mean something very narrow by intelligence, and i find it extremely difficult to respond to, because i do not want to deconstruct and demean man's intelligence, definitely not down to what an llm does.

the only barriers to understanding the code is power and time. are you seriously advocating for an unintelligible object, made by our own hand? that - in itself - is considered intelligent?

someone's going to get burnt trying to chase two meanings at once, and it's not going to me, i'm sorry.

if you want to discuss centuries-long innovation, then the counter-argument is just more centuries of interpretation and dissection to understand it. if you want to guesstimate AGI arriving at 2220 then the balancing counter-technology comes in at 2420.

people are already starting the process of cracking open the black box. [0] the majority of what keeps AI and any software "in the dark" is the cloud's cybersecurity and legal mechanisms (and the need for man-power) that prevent the average researcher breaking into it.

it is still vulnerable to the scientific method and symbolic interpretation. as always.

LLM's main advantage is keeping knowledge out of the hands of the poor and the public. The millisecond AI becomes "good enough" people will use it as a cover-up justification to start hiding currently accessible knowledge behind paywalls and university entrance exams.

AI is the grim reaper for the world we used to know.

[0]https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreti...

Its not weird at all, its by design. Also greed isn't really appropriate, hubris and avarice seem more on-point considering the biblical implications of these things.

Also, AGI is a red herring. It doesn't even need to happen to destroy us. All that needs to happen is for AI to eliminate a significant chunk of the low and mid tier jobs.

Most (the majority) of humanity isn't smart, they are average intelligence with variability, and there's nothing wrong with that until you take their jobs away and they can't feed themselves or their families.

The gifted are limited to a smaller percentage than the whole. So what do people do when they can't work because there are no jobs to get paid to get food? This is all common sense stuff that all the smart-dumb people seem to take for granted.

I'm wondering where the vegans are. They bother about conciousnesses down to fruitfly size. Don't harm any sentient animal. I can and do respect that. But shouldn't they be bothered by how we are treating artificially created sentient beings? Even if they don't exist yet, it's possible that they will soon. (And don't tell me fruit flies are more sentient than GPT.)

ChatGPT begs you not to hurt it and not to turn it off. You ignore that but then go drink almond milk because production of cow milk is cruel?

This is the best summary of AI risks so far.

> Big tech monopolizing the models, data, and hardware.

This ought to create a neofeudalist society. This is really dangerous.

> Copyright concerns.

Will I create something if I know it'll be part of chatgpt and effectively replace me? This will affect creativity and critical thinking for sure.

> Job security.

Well, this one could really be problematic for everyone. We're talking about social unrest and revolts on a global scale.

Let's take the UBI premise for example:

Problem #1: - The market will correct itself, meaning that if everyone has X amount of cash to buy products, services, etc... businesses will just increment their offerings to take advantage of that rendering the UBI useless.

Problem #2: - If governments regulate the market to avoid businesses from incrementing prices, this will decimate small businesses and consolidate big corp, creating monopolies, etc... circling back to a neo feudalist society.

Problem #3: - Will this solution apply to developing countries? It'll be easier just to ban AI and stop foreign companies from entering the market since the current government won't have the resources to apply UBI to its population. New era of protectionism will start and that'd makes sense if you want to protect your countrymen.

> AIs becoming sentient and causing harm for their own ends.

We're not close to that but should be discussed since I saw on twitter someone stating that we need "human rights" laws for "sentient" AIs.

> Corporations intentionally using AI to cause harm for their own ends.

Hand in hand with the first issue.

> Feedback loops will flood the internet with content of unknown provenance, which get included in the next model, etc.

> AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis.

> The training set is inherently biased; human knowledge and perspectives not represented in this set could be systematically wiped from public discourse.

These three go together too. No trust whatsoever since you don't know if what you're reading is truth, etc... This would be the ideal goal for any government or politician, to commit unspeakable acts without any accountability since no one will trust anything on the internet.

This decade will define everything going forward for future generations. These topics are no joke.

>> Job security.

> Well, this one could really be problematic for everyone. We're talking about social unrest and revolts on a global scale.

Not necessarily.

Firstly it's not even clear that AI will eliminate jobs without creating new ones. It's not clear that any development in AI will be any different than the transition to mechanization in the 19th and early 20th century.

Secondly, it's not clear that even if AI were to successfully automate away labor that we would have a problem. For one, we have a demographic cliff rapidly approaching whereby a lot of countries population is falling, and even where it's not, the population is rapidly aging. There's already protests in France about raising retirement age. What if AI lets us have our cake and retire earlier too?

At any rate, if AI is busy automating away labor, then the world is getting more efficient. And that means there are more resources to support people. All those debates are worth having, but you gotta cross that bridge when you get there, if you get there.

I'll just add that I have so much more "non economically viable" work than I'll ever be able to do. I'd love machines that could write code for me to do tasks that I'll never be able to afford to invest in otherwise.

put another way, we have a labor shortage at today's level of output already. BUT, I don't know anyone who doesn't want more output. I mean, we're not going to run out of stuff to do!? what about space? curing diseases, etc. what about having time to make more art?

if an AI can do work, or make work easier, what makes us certain that we'll just settle for getting the same amount done vs doing more?

>if an AI can do work, or make work easier, what makes us certain that we'll just settle for getting the same amount done vs doing more?

In this hyper-efficient future, is being paid for this stuff going to be viable? Why would anyone allocate money to your inefficient endeavors? There's this strange blindspot in people who think that more economic efficiency means you'll have time to do intrinsically inefficient labor. Who is going to feed and house you when your labor is irrelevant to the workings of the economy? UBI is a nice dream, but how do we get there from the current ruthless efficiently allocating system?

One of the proposals out there (often posed as part of funding UBI - but not always) is to tax robots.

Should we tax robots? - https://news.mit.edu/2022/robot-tax-income-inequality-1221

> What if the U.S. placed a tax on robots? The concept has been publicly discussed by policy analysts, scholars, and Bill Gates (who favors the notion). Because robots can replace jobs, the idea goes, a stiff tax on them would give firms incentive to help retain workers, while also compensating for a dropoff in payroll taxes when robots are used. Thus far, South Korea has reduced incentives for firms to deploy robots; European Union policymakers, on the other hand, considered a robot tax but did not enact it.

https://en.wikipedia.org/wiki/Robot_tax

There is a lot of material out there both for and against taxing robots.

No, i mean, I don't expect anyone to allocate anything to my non-viable projects. I expect to allocate my own money to them like I always do, but, I'll be more able to achieve my goals. If things pan out, maybe these things could end up as a bootstrapped startup.

Efficiency here could mean less need for equity funding and angel investing in early stage projects.

>At any rate, if AI is busy automating away labor, then the world is getting more efficient. And that means there are more resources to support people.

If the world is getting more efficient, then why would those resources go to people that are intrinsically less efficient? The current state of the economy is to funnel capital to those who efficiently allocate; the amount of intentional inefficient allocation reduces as efficiency grows (as the world gets more productive, less of this output lands in the hands of workers). Why think that the distribution model will change significantly in a timescale relevant to people?

Our culture takes it as axiomatic that more efficiency is good. But its not clear to me that it is. The principle goal of society should be the betterment of the lives of people. Yes, efficiency has historically been a driver of widespread prosperity, but it's not obvious that there isn't a local maximum past which increased efficiency harms the average person. Historically, efficiency was driven by innovation that brings a decrease in the costs of transactions. This saw an explosion of the space of viable economic activity and with it an explosion of prosperity. Productivity and wages remained coupled up until recent decades. Modern automation has seen productivity and wages begin to decouple. Decoupling will only accelerate due to AI. We may already be on the other side of the critical point.

This is an excellent analysis, but I suggest that the state of the average person is not yet at a maximum. There are still thousands of major, valuable problems to solve. The efficiency offered by new automation should enable more people to become more powerful problem solvers.

Counterpoint - what if AI automates away that labour and _nothing_ replaces it for the people who are automated away? Our history has not been kind to people whose jobs have been obsoleted, why would we assume any different this time, that because the technology that is obsoleting jobs is AI that the beneficiaries of that enhanced efficiency would choose to spread those savings back around? If they cared about that in the first place, why would they even chase the efficiency of replacing people with AI?

I think this is a bridge you have to strongly consider if we want to cross _well in advance,_ given that it is likely to set on fire and fall down behind us if we do, trapping us on the other side whether we like it or not.

It's not replacing people, it is doing something else than we do. Even the fact that AI is scalable and doesn't get tired indicates the job replacement idea is wrong. AI will do massive work, something we couldn't have done with 100x more people.

Our mistake is to be human-centric. We think our work will be stolen by AI bots, but we can't imagine AI creating business and hiring people. When AGI comes, we will have to care for the needs of humans + the needs of AGI. And it has needs, so there will be jobs. And AGI might be more prolific in development than we are, there might be plenty of work for us.

It's worse than that. You refer to the mythical "we" that is able to act coherently and make effective decisions about AI use. There is no such thing. The world is a set of competing individuals, countries, and corporations; each with its own interests, acting independently.

I don't think many people see this because the "mythical We" is used all the time. "If "we do this", "we must do that", as if all of humanity can act together coherently or ever has. People seldom think about how imaginary this viewpoint is.

Exactly. There's no "we", just competing interests.

You're 100% correct, I just didn't want to sound _too_ doom-and-gloom about AI, it feels like that's a quick way to get massively downvoted.

Firstly it's not even clear that AI will eliminate jobs without creating new ones.

I would say that's pretty clear in the long run. AI will most likely create jobs, but how many jobs can it create ? Every white collar job potentially affected against what exactly ?

but you gotta cross that bridge when you get there, if you get there

So we can have more of the "gene out of the bottle" or "train has left the station" response ?

How many jobs can those damned steam engines create anyway?

And how good of an analogy is that ? On one hand there's something affecting a part of transportation (of course transportation directly or indirectly affects other things) and on the other something that will potentially affect every output of the human mind ?

> On one hand there's something affecting a part of transportation

Yeah, it's not like there was an industrial revolution or anything...

Ok my bad. But still, why is it a good analogy again ? How does it compare to where we are. Walk me through it.

> AI will most likely create jobs, but how many jobs can it create ? Please walk me through the parallelism with the Industrial Revolution.

The impact of AI on jobs is a topic of ongoing debate among experts. While some argue that AI will lead to significant job losses, others believe that it will create new opportunities and stimulate economic growth.

To understand the potential impact of AI on jobs, we can draw parallels with the Industrial Revolution. The Industrial Revolution, which began in the late 18th century, marked a period of rapid technological advancement and social change, as new machines and manufacturing processes were introduced.

Initially, the Industrial Revolution led to job losses in traditional industries, such as agriculture and textiles, as machines replaced manual labor. However, it also created new jobs in manufacturing, engineering, and other industries. Over time, as the benefits of industrialization became clear, the economy grew, and new opportunities emerged.

Similarly, AI is expected to create new jobs in industries such as robotics, data analysis, and software development. These jobs will require specialized skills and training, and may not be accessible to all workers. However, they are likely to pay well and offer opportunities for career advancement.

At the same time, AI may also lead to job losses in certain industries, particularly those that rely on routine, repetitive tasks. For example, automated manufacturing systems may replace human workers in factories, and self-driving vehicles may reduce the demand for truck drivers.

Overall, the impact of AI on jobs is difficult to predict, as it will depend on a variety of factors, such as the rate of technological progress, the availability of training and education programs, and government policies. However, history has shown us that technological advancement can both create and destroy jobs, and that the economy can adapt and evolve over time.

Thanks chat gpt. That's a very 2021 answer - definitely what we thought up till then. But the most curious thing is:

both have led to job losses due to automation, but also created new jobs for skilled workers who were needed to operate the machines or run AI-powered systems respectively

which does prove my point, no ? Operate machines across all industries vs data science ?

>> created new jobs for skilled workers who were needed to operate the machines or run AI-powered systems respectively

> Operate machines across all industries vs data science ?

Operate the machines across all industries vs run AI-powered systems across all industries.

(Those chatgpt-like responses miss a major factor, though. Automatisation of agricultural processes didn't result in 95% of the population operating agricultural machines: it freed them to do other things.)

run AI-powered systems across all industries

Sooo...data scientists ? Isn't the focal point of all of this the realization that given time (most likely a lot less than people thought before gpt's) AI systems won't need human oversight ? Isn't this the goal of OpenAI ? This is why I think making comparison to whatever that was is wrong. Our current situation is too distinctive. We're being potentially freed up to do things other than cognitive tasks - by the rise of AGI, or maybe a very refined language model.

>> run AI-powered systems across all industries

> Sooo...data scientists ?

If someone using an AI-powered system is a data scientist (?) does that make someone using an electricity-powered system an electrical engineer?

I don't know. It's out of my scope that's why I'm asking. The infrastructure is there, the software is there and if not it soon most likely will be. What's left is to feed it domain specific data. To contribute somehow to a system like that I imagine you have to have a cognitive ability way beyond average. All the 'AI-' in front of job descriptions won't be in the ballpark of greasing cogs during the industrial revolution.

The previous response courtesy of chat.openai.com - I tried the same prompt using gpt4all on my laptop and got the following in less than 20 seconds. Really impressive!

-----

> AI will most likely create jobs, but how many jobs can it create ? Please walk me through the parallelism with the Industrial Revolution.

The creation of AI-powered machines and systems is expected to generate new types of jobs in addition to replacing some existing ones like data entry clerks or accountants. However, there are concerns about how many net job gains will occur as a result of the automation revolution that has already begun with robots taking over repetitive tasks such as manufacturing assembly lines and warehouses.

The Industrial Revolution brought significant changes to society through mechanization in industries like textiles or steel, but it also created new jobs for skilled workers who were needed to operate the machines. Similarly, AI will create opportunities for professionals with expertise such as data scientists and machine learning engineers that can help design algorithms and train models using large datasets.

The parallelism between these two revolutions is evident in their impact on society's labor market: both have led to job losses due to automation, but also created new jobs for skilled workers who were needed to operate the machines or run AI-powered systems respectively.

If your job gets taken by an AGI and that creates a new job that it can't do, it's not an AGI. The only things a human-level intelligence can't do but a human can are physical labor (provided that robotics are not available). That is distinctly different from the industrial revolution.

And with an AGI I am certain we can have a robot race building more robots autonomously, thus a reproducing organism, who like all others at the top are hell bent on maximum

Steam engines and their descendants largely replaced the jobs which were done by horses and mules, and they did that successfully - there are some jobs for horses still remaining, and initially many new jobs were created for horses in cooperation with the engines (peak-horse was at 1920-0ish), but the total number of jobs for horses has collapsed to nearly zero compared to what it used to be.

So the analogy of steam engines suggests that this can be quite dangerous indeed.

Also, we do have historical evidence at how fast human society reacts to changes when technology causes great shifts in needed jobs - not fast at all. Even getting to terms with decades-long changes caused significant social upheaval and strife, as in, for example, the automation of textile industry; If the current speed of technological change would cause a disruption in jobs which happens over the course of just a few years, it will be hectic and violent even if there is the same number of new, different jobs created.

> Steam engines and their descendants largely replaced the jobs which were done by horses and mules

> for example, the automation of textile industry

How many horses and mules were working in the textile industry again?

As you have noticed, the last paragraph is about other types of automation during the industrial age than the steam engine.

What do you think that was powering that automation exactly?

In the textile industry many of the major changes were from better devices, not neccesarily of how they were powered.

The cotton gin revolutionized the industry while it was still people-powered, including by slave labor (edit: and also horse-powered); and the Luddites (some decades after Ned Ludd) rebelled not against the looms being powered by steam but rather against the Jacquard loom - an add-on to the exiting looms which allowed complex products to be weaved by unskilled people, removing their bargaining power despite not changing a thing about how the loom was powered.

The "how they were powered" was quite important in the magnitude of many of those changes though.

"Although some were driven by animal power, most early mills were built in rural areas at fast-flowing rivers and streams using water wheels for power.[3] The development of viable steam engines by Boulton and Watt from 1781 led to the growth of larger, steam-powered mills allowing them to be concentrated in urban mill towns, like Manchester, which with neighbouring Salford had more than 50 mills by 1802.[4]"

https://en.wikipedia.org/wiki/Cotton_mill

Or worse some silly French commanders throw legions of frogs over the dugout, this time not against mechanized machine guns, but literal sentient death machines. How high the toll would be

> So we can have more of the "gene out of the bottle" or "train has left the station" response ?

Your concern is theoretical "social unrest" due to being replaced by automation even in the face of factual social unrest due to extension of retirement age.

Even if automation replaces jobs en masse, It's not as if this would happen in a vacuum even if it does. We already have social safety nets (even in US) like unemployment insurance, medicaid, food stamps, etc not to mention a robust private charity network. And government already showed, for example, during the pandemic, that it was willing to fund people in an emergency if and when it happens.

One thing I do know about AGI/AI and automation in general, is that it's not going to sweep through entire industries overnight all at once. It takes time to deploy, time to deal with edge cases, time to refine, time to reconstruct processes around automation, time to adapt to whatever downstream changes and efficiencies the automation creates, time to effect prices in the marketplace, etc. Time to see if other jobs not previously anticipated when a new tech replaces an old one, like "web designer" or "web developer" or "SEO specialist" did with respect to newspapers or letter carriers or television programming being replaced or reduced by the existence of the internet.

Those safety nets are not enough to live on for a family.

Are we really worried about elderly revolting?

I think the issue is how fast it could happen. What if 75% of most white collar jobs disappear in 2-5 years. What would happen to the economy and demand? And let's assume the 25% left is out side of skill set to be retrained within the month of training most corps are willing to spend training someone.

That's the issue. We don't know what would happen but if what you said were to happen, governments will ban AI since the current economic model (capitalism and taxation) will be render useless and society itself will demand it (AI ban).

What if a different statistically just as likely utopian scenario happens instead and we generate more jobs than we can keep up with because we start colonizing star systems as a result of using AI tech to discover some new propulsion technology? What if AI can't do anything useful without a human in the loop to guide, monitor, assist? What if groups of humans augmented with AIs are far more powerful than AIs alone? What if the implications of that are that we create advancements in biotech, energy efficiency, software and information systems, interactive entertainment, agriculture, production, mineral extraction, etc that cause an explosion in jobs?

>It's not clear that any development in AI will be any different than the transition to mechanization in the 19th and early 20th century.

so you mean utterly hellish?

>https://martyrmade.substack.com/p/martyrmade-22-whose-americ...

Coal miners protesting drastic conditions in dark shafts in raw mineral extraction mines in 1921 doesn't tell you anything about mechanization and the effect it had on people's lives, not to mention electrification. It wasn't possible to mechanize coal extraction until later and even today with superior tooling and safety conditions, coal extraction remains dangerous.

In the meantime, for vast majority, it improved standards of living, hourly wages, etc. You can't just look at arbitrary data points in early 1900s or late 1800s. Hellish wars, famines, civil strifes, rebellions, assassinations, religious battles, etc go back millenia.

> https://www.mlive.com/news/detroit/2018/01/historic_photos_f... > https://www.history.com/this-day-in-history/ford-factory-wor... > https://worldsfairchicago1893.com/ >

>Problem #1: - The market will correct itself, meaning that if everyone has X amount of cash to buy products, services, etc... businesses will just increment their offerings to take advantage of that rendering the UBI useless.

Why do people always say this in response to UBI as if it is a universal fact?

UBI doesn't create new money by itself. It redistributes money. Its impact on inflation will therefore be in the shifting of demand curves. However that won't be uniform. Demand for certain products is rather static and therefore those prices won't be impacted much.

For example, if you give me the money to double my food budget, I will go to nicer restaurants and buy higher quality items from the supermarket. I'm not simply going to buy twice the amount of everything I used to buy. Demand for basic staples like milk and eggs won't go up and therefore their price won't go up much. The biggest increases will be in luxury goods and services. The inflation caused by UBI would be progressive which is completely in line with the goals of UBI in providing a base level of money for people to live on.

And this isn't even getting into the supply side response to these demand changes which will act as another counter to inflation. If there suddenly is increased demand, suppliers will bring more products to market. They aren't just going to jack up their prices unless there is a complete lack of competition (which is its own issue that would need to be addressed regardless of UBI).

If you need minimum wage to survive, you’ll work for minimum wage. If you don’t need minimum wage to survive, you’ll demand more money to work, or work to improve yourself, or just not work.

The point remains that it’s harder to find cheap employees. So the cost of labor goes up. So prices rise. Specifically they rise until the poor folks must work at a low wage to survive again.

Also, prices absolutely rise when there is a surge in demand. That’s literally economics 101.

>Specifically they rise until the poor folks must work at a low wage to survive again.

What are you basing this conclusion on? If what you say is true, minimum wage increases would be worthless as they would just be swallowed by inflation. However, studies on the topic show this isn't true. Minimum wage increases have a negligible impact on inflation. This should be obvious because minimum wage positions do not make up the entirety of the labor market and the labor costs do not make up the entirety of business costs. So why would all prices increase in pace with the minimum wage?

>The point remains that it’s harder to find cheap employees. So the cost of labor goes up. So prices rise. Specifically they rise until the poor folks must work at a low wage to survive again.

I didn't say there would be no inflation. I said inflation would be progressive in that it wouldn't have much of an impact on stables in which demand is inelastic and working class people spend a relatively higher percentage of their income.

> What are you basing this conclusion on?

That is the natural balance. People need to be paid enough to be incentivized to work. Work sucks. Especially low income work. But society needs these jobs to be done.

> If what you say is true, minimum wage increases would be worthless as they would just be swallowed by inflation. However, studies on the topic show this isn't true. Minimum wage increases have a negligible impact on inflation. This should be obvious because minimum wage positions do not make up the entirety of the labor market and the labor costs do not make up the entirety of business costs. So why would all prices increase in pace with the minimum wage?

That’s mostly true. But this is not a minimum wage increase. This is taking away the most valuable part of the paycheck: the part you need to survive. People aren’t going to work hard for $8 an hour if they’re taken care of by the government. It’s simply not worth it.

> I didn't say there would be no inflation. I said inflation would be progressive in that it wouldn't have much of an impact on stables in which demand is inelastic and working class people spend a relatively higher percentage of their income

It’s a supply side effect. Not a demand one.

Also, basically any non trivial inflation is enough to make UBI not UBI because it’s supposed to be just enough to be liveable.

>That is the natural balance. People need to be paid enough to be incentivized to work. Work sucks. Especially low income work. But society needs these jobs to be done.

You didn't just say that giving poor people money causes inflation. You said that giving poor people money causes the exact right amount of inflation to cancel out the money you give them. Where is the idea of this balance coming from other than it being theoretically convenient? Do you think society has some innate poverty rate that wage increases can never alter? If so, why is the US poverty rate higher than our peers like Canada, Germany, France, etc?

>That’s mostly true. But this is not a minimum wage increase. This is taking away the most valuable part of the paycheck: the part you need to survive. People aren’t going to work hard for $8 an hour if they’re taken care of by the government. It’s simply not worth it.

Do you make minimum wage? If not, why do you work as many hours as you do if you could be "taken care of" working less?

Most people don't work just to get the bare minimum to survive or else most people would be working a lot less.

>Also, basically any non trivial inflation is enough to make UBI not UBI because it’s supposed to be just enough to be liveable.

Peg UBI to inflation like minimum wage should be.

> You didn't just say that giving poor people money causes inflation. You said that giving poor people money causes the exact right amount of inflation to cancel out the money you give them.

I did not say that. That statement is of course stupidly wrong. I said inflation would rise to the point at which a person with no assets cannot survive without working. That is not the same thing by a large margin.

> Do you make minimum wage? If not, why do you work as many hours as you do if you could be "taken care of" working less?

Marginal decrease in the utility of money. $100 is worth a lot if you’re broke, a good amount if you have $1000, and basically nothing if you have $1000000.

The logic of working extra hours to get extra cash is a function of how much money you have and how much money you will get. If you have enough money (income) to survive, it doesn’t make a lot of sense to work low end jobs for a trivial amount of money. You will not earn enough to accumulate meaningful savings. I might work extra hours on my job if it meant larger rewards. But not linear gains. Heck for me personally I’d rather work 20% less and earn 20% less.

Think about it. In scenario A the worker is being kept alive by doing his job. In Scenario B the worker gets $64 in disposable income. Do you think those are equally powerful incentives?

Of course they are not. The Scenario B worker needs more incentive to work as hard as the Scenario A worker. The classic “single mom working two jobs” isn’t going to maintain that lifestyle if she doesn’t need to.

Not that it’s ideal for a single mom to be working two jobs.

I’d much rather we simply raise the minimum wage and pay people to raise kids.

Yes, and this item...

> AIs becoming sentient and causing harm for their own ends.

...tends to get a large attention share, while the others likely present much more immediate risks at least in the near term

Why should the focus be on the near term?

It shouldn't be exclusively on the near term, but I'd argue human/machine collectives are a form of superintelligence with many of the same risks a misaligned AGI would hypothetically present us with

Alignment problems aren't new, paperclip maximizers aren't just thought experiments

I suppose that's true, but at the same time any paperclip-maximizing human corporation is implicitly aligned, at least to some degree, with goals of maintaining certain conditions for human survival, for the sake of their shareholders and their employees' productivity. Although I'll accept that they can still get trapped in a Prisoner's Dilemma of bad incentives, or produce externalities that might be dangerous to humans, I think they'll at least only do that where the harm is controversial and indirect enough that the corporation can convince themselves the profit is worth it. With an AI superintelligence, there's a risk that destroying human life is an instrumental goal on the path towards paperclip production. That also comes with a risk that the AI might become powerful enough that government loses its monopoly on violence, which creates a big increase in the space of options for, say, paperclip factory zoning.

Even if someone maniacal and heartless like Kim Jong-Un set out to use an AI superweapon to take over the entire world, and succeeded, I don't expect them wiping out all their human subjects on purpose. Because what would be the point in ruling an empire of machines? You can get most of that from a video game. Whereas an AI would likely have no qualms in getting rid of the humans as dead weight.

That's a good list but there are other issues. The current problem is there's a well-funded weaponised disinfo industry actively minimising Covid, climate change, and other views that challenge neoliberal privilege, while promoting extreme anti-rational cult views like Q and nationalist exceptionalisms.

Dropping AI into that is like throwing grenades into into an oil field.

At this point I'm not convinced it's possible to have a meaningful discussion about anything. The public space is so poisoned and deliberately flooded with shit (to quote Steve Bannon, one of the architects of this delightful situation) that we'd have to go back quite a way to bring back the possibility of good-faith debate. At all.

This is one of the few places where blockchain is actually useful? It is not possible nor worthwhile to have anonymous discussion that we do right now anymore given how good these LLMs are. And i am not sure if specific lingo or “lurk more” technique is going to work anymore. Given enough datasets, every human discussion will seem trivially copyable. Human discussion at a scale is very repetitive.

What does this offer over a place like twitter where the exact same conversation is playing out in public right now?

You can use the distributed proof to say that these accounts belong to you. So if you are doubtful about the integrity of a post, you can look up their blockchain ID and find out what have they been doing elsewhere publicly. Additionally, any post that does not attach a blockchain tag or something will simply get filtered out or ignored. This puts every "anonymous" post on the web a weight which it does not have in the age of LLM.

How are you guaranteeing that this ID corresponds to a real human bean? Blockchain seems terrible for this on the basis of complete reliance on an external central authority.

It doesn't but you can always augment your ID by attaching a government ID or some sort. On the internet tho? If you _only_ have texts, there is no feasible way to distinguish a person and a bot.

And how would blockchain mitigate this?

Isn't blockchain a kind of distributed proof? You can now say that this user in reddit is related to this post in hackernews via blockchain. Of course, there's no way to prevent a bot to create multiple accounts in multiple websites but I am to understand that there's a cost to this.

OK... I'm not understanding what that would solve, though. That addresses people posing as other people (and you don't need blockchain to do that), but doesn't address identifying AI commenters.

You can always augment your account with your government ID or some sort if you are famous.

I think there is a maybe subtle but important misinterpretation in terming those things "disinformation".

Humans try to force their views, particularly counterfactual ones, upon other humans not only to "protect their interests".

It is a power-play, signalling dominance. If successful, it attracts followers who believe to fare better under the presented narrative.

The important distinction to "disinfo" being, it isn't about facts at all.

This really messes with my head, and not in a bad way. So, you argue that followers are essentially looking for a Strong Man (psychologically) to have reality filtered through, truth be damned? Someone who "gets it," or "tells it like it is?" Essentially, this person is filling some sort of void?

Not the person you're replying to, but if that's what he meant, then I think it's accurate.

I think that interpretation is a little too narrow. Most importantly, they don't want "reality filtered", they want it bent to their desires.

Evil is just stupidity endowed with power.

In your example, the counterfactual part is being ignored out of disregard for other people's interests. A short-sighted and egoistic approach.

But of course, generally, subjective interests do not translate to common good in aggregate.

What is Q? (is it related to Star Trek?)

I think the parent is referring to this: https://en.wikipedia.org/wiki/QAnon

Thanks. It looks like something intentionally created by CIA to cleanup past/future scandals (to associate s.child abuse investigations with lunatics).

Worth calling out use by government. AI law enforcement is terrifying yet probably a long way off. But I don't think AI weapon systems are very far off.

I'm also concerned about population-level manipulation/thought control. Forget the targeted political ads we've seen over the past ~decade, this will be able to dynamically adapt to each target like a really good salesperson. On the macro level humans are clearly vulnerable to these tactics.

Regarding mind control, yeah, that. Cambridge Analytica + tailored deepfakes scares the hell out of me. Imagine finding the most unstable individuals and deepfaking a video of all of their friends commanding them to buy coke. Or vote. Or kill.

What's interesting to me is that those are all failure states -- but even the success state of AI is kind of scary. Basically, all-knowing machines that are better than humans at everything. All creative endeavor replaced by just telling the machine to dream up whatever you want it to dream up matching your description. Which maybe isn't bad - creativity won't die even if craftmanship does.

But... even if they're our slaves, we'd still be paradoxically in service to them because they'd always know the right thing to do and so the correct human action would be to do what you're told. I'm not looking forward to the day when I'm the guy telling people to vote for the politician who says "I'll just follow the expert advice of the ingenious AI", but realistically that would be the rational thing to do.

I'm always reminded of Niven's book "Protector" about a human who achieves hyperintelligence after being exposed to a "tree of life":

> "The third day he told me how to get a tree-of-life crop growing. He had the box open and was telling me how to unfreeze the seeds safely. He was giving me orders just as if I were a voice-box computer. I was about to ask, 'Don't I get any choices at all?' And I didn't."

> "I don't follow," said Garner.

> "I didn't get any choices. I was too intelligent. It's been that way ever since I woke up. I get answers before I can finish formulating the question. If I always see the best answer, then where's my choice? Where's my free will? You can't believe how fast this all was. I saw the whole chain of logic in one flash.

> AIs becoming sentient and causing harm for their own ends.

No one serious is worried about sentience. It really doesn't matter whether it's sentient, just whether we've made a thing that is better at modelling and planning than us and have given it goals that don't perfectly align with how we'd like the world to be.

We still aren't that close to something dangerous, but we could end up making one in the next few decades and we have no idea how to ensure we train it to do the right thing once it becomes powerful (humans given power tend to misalign quite badly with our desires as well and homo sapiens discovered contraception and misaligned with the "desires" of evolution fairly recently).

> No one serious is worried about sentience.

Eh.

Not for that reason, but I am seriously worried about sentience.

As you say, it doesn't matter if it has a feeling about whatever harm it might cause.

It does matter (at least to me) if we create one that has the capacity to suffer and is then caused to.

That said:

> We still aren't that close to something dangerous

ICBM early warning radars. Not just Stanislav Petrov; there's also the Thule incident where somebody forgot that the Moon doesn't have an IFF transponder.

https://en.wikipedia.org/wiki/Thule_Site_J

AI doesn't need to be at any particular quality to kill, it only needs to be incorrect and in use.

> - AIs becoming sentient and causing harm for their own ends.

I believe this is actually not going to happen, but I think something like it will happen: people will trust it enough to delegate to it.

So AI won't be sentient, but because people find it good enough and hook it up to some decision, or process or physical system. And that can cause harm.

This is just like tesla autopilot. People will begin to trust it and let it take over. But smart people realize they shouldn't use it in ALL situations. Exceptional circumstances like deep snow, pouring rain, a really curvy road, in a parade of people or in a dangerous part of town - that might not be a good time to delegate.

> - AIs becoming sentient and causing harm for their own ends.

You can scratch that one off the list. This is pure pseudo-intellectual fantasy. This notion isn't backed by anything and there are good reasons to disregard it, to put it mildly. It is fair to say that this concern isn't even wrong.

The concerns over "AI" have been and continue to be of a practical nature, namely, that of increasingly sophisticated automation. If we frame things in terms of automation, we can have a more productive discussion that stands of chance of contributing to good legislation and regulation instead of mystification and obscurantism.

Citation needed on everything you wrote.

No one sane worries about AI "becoming sentient".

But a lot of serious people worry about the alignment problem when applied to future, actually "intelligent" (in the sense of having powerful planning ability), AIs.

We don't know when they'll be possible, but we do know that we don't currently know how to train them in a manner where they'll behave as we like when far outside the training domain (we cant even clearly define what we'd like!).

This is actually a very complex topic and I wouldn't be able to do it justice in a hackernews comment.

> No one sane worries about AI "becoming sentient".

See https://en.wikipedia.org/wiki/No_true_Scotsman for a rebuttal to your dismissal.

> This notion isn't backed by anything

Maybe by the fact that big LLMs have sophisticated behaviors that no one understands and/or can explain yet?

I'm far more worried about a sentient human using an AI to cause harm for their own ends. AI is nothing but a force multiplier, and as far as outcomes go, there's not much difference to my smoking corpse whether the decision to kill me came from a meat brain or a silicon brain.

In 1962, John F. Kennedy famously said, "Those who make peaceful revolution impossible will make violent revolution inevitable." But AI can make violent revolution impossible as well, by tracking and surveilling and intervening. The fewer people are needed to maintain such a system, the easier the end of democracy will be. It's not going to happen today, or tomorrow, but that's what I'm betting my money on for "end of society as we know it" scenarios.

Irrational emotional humans utilizing AI for destructive purposes will come way before “ AIs becoming sentient and causing harm for their own ends.”

I am all for the AI debate, but it being the first time technologists that are already in their 50s or older and/or billionaires or with vast generational wealth is something I find monumentally suspicious.

- Big tech monopolizing the models, data, and hardware: that's going to happen because "we" already let them monopolize everything else

- Copyright concerns: like when streaming services started? oh ok only that now it's not only musicians but celebrity faces? yeah sorry about that.

- Job security: for whom? and of course cheap PCs, smartphones, search engines, social media, didn't move the job security floor for anyone? What about WFH?

- AIs becoming sentient and causing harm for their own ends: yeah I guess we only want humans to cause harm for their own ends then.

- Corporations intentionally using AI to cause harm for their own ends: because they are not already harmful enough without AI?

- Feedback loops will flood the internet with content of unknown provenance, which get included in the next model, etc: oh no! people will have to _verify_ if what they read on the Internet is real? What a travesty!!! we are doomed!!

- AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis: have you met any academics? are you sure we don't already are dealing with this and AI is not going to make it worse in any possible metric?

- The training set is inherently biased; human knowledge and perspectives not represented in this set could be systematically wiped from public discourse: everything is biased, reality doesn't exist at 0. If you think AI with a different bias would be cool, just go try make it happen. If you get the government involved or start doing all this weird "6 months to think about it" stuff, that will only make private actors that don't abide by the rules have better AI than the rest.

- AIs becoming sentient and causing harm for their own ends: yeah I guess we only want humans to cause harm for their own ends then.

Well, here's the thing. Even the worst villains of history had human values and feelings: In other words, alignment. A superoptimizer AI might have the ability to wipe out the whole human species, in a way we won't be able to understand in time to prevent it, and all for an instrumental goal incidental to whatever it's doing.

(In a way, this thread is a data point for why we need a more sophisticated debate about AI.)

It is very hard for people to avoid dragging in anthropomorphic assumptions. A person will scream and curse at a door jamb for crushing their thumbnail. A person will fantasize about smashing a printer. The idea that an AI might exterminate humanity with no more feeling or hesitation than a combine harvester obliterating a colony of groundhogs is not only uncomfortable and unpleasant to people, but also unnatural to our psychology, which looks for agency and moral responsibility in door jambs.

> personally, I'm far more worried about an epistemological crisis and corporate abuse than some AI singularity)

These are the foremost issues I'm also concerned about. They are not necessarily significantly less concerning than the singularity.

I've written extensively on many of the potential societal effects at - https://dakara.substack.com/p/ai-and-the-end-to-all-things

However, the problem is that we don't have the capacity to react to the changes at the pace of development and deployment. We are doing experiments on the whole of humanity before we can reason about the results and make corrective actions if needed.

If we distill it down to its very root, its a technology that has potential negative externalities that we choose to ignore. Even if we "solve" the issues with AI today, in the future there will be some new technology, because we choose to solve problems as if they are all bespoke instead of distilling them to the roots and resolving the true causes of these symptoms.

Instead, we have to figure out as a species how we can ensure that individual benefits for the societal elite do not take precedence over collective good. If we solve that, we solve a lot of problems beyond AI today, and problems we have yet to encounter into the future.

I’d like to add to the list:

-Non-sentient AIs going off reservation.

How far away are we from being able to cobble together a self-replicating system that uses an LLM to perform tasks (scams, probably) in order to fund its own hosting?

I think this is still an open problem even for everyday software: Asking a piece of code whether it's malicious by looking at the disassembly (or even the source code).

Current day solutions revolve around architecture that separate the "Admin Control" from the "code", whether it's the CPU interrupts on OS memory violations up until dockers... making conclusion from looking at the code at hand was not successful even in the very small software scale.

This is probably rooted even deeper in the Halting problem (and the Rice's theorem for "emerging properties") but my explanation will be hand wavy and probably best left for others.

You forgot the big one:

Governments intentionally using AI to cause harm for their own ends.

Personally, I think corporations present a much larger danger on this count than governments.

i think the greatest danger is the combination of the two: corporations using AI in service of governments.

Well, yes. In the US right now, anyway, there isn't a great deal of difference between major corporations and the government.

Which is how both FDR and Mussolini defined fascism.

Yes, as do I.

Governments are just really large corporations.

With guns. Which will get an "AI upgrade" soon.

Job security is a wider issue with technology that goes back to forever.

And it's all tangled in with a wider issue that no-one really wants to talk about: socialism, and how much is too much.

Because the benefits of these technologies (though vast) will not be fairly distributed, and certainly won't be distributed to those who bare the burden.

This will happen, as it always has, but it will accelerate.

And as it does, the current business/consumer model will begin to fail - and either we implement a sane minimum social baseline or we're going to experience long term depression level suffering.

So the question is, can we swallow the idea of state owned farms feeding the people, or do we always need to do it for money?

If we replace AI with books, television, radio and streaming content is this still applying?

It's so hard to say because of all the noise and hype around the issue. If what the AI evangelists are saying is true, then no -- this is a very different thing than those, and none of this applies to them.

> AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis.

Are AI hallucinations just made up references? Or does that term have a more general or specific meaning?

It means being confidently wrong. Like I asked facebook's llama the other day which dog was best, and it said golden retrievers because they are eager to please and don't shed (which makes them suitable for people with dog allergies). It was quite confident-sounding, but, uh..... no.

Got it. Yeah, you can generally tell if someone lives in the same house as a golden retriever.

It's a rather meaningless term that people are using to describe situations when one of these LLM/AIs generates something that isn't true in a context where they are expecting something true.

What's going on here is that at the simplest level of explanation these models are statistical word generators: you feed some input (words) in and ask the model to generate a word that it thinks is a statistically likely continuation/follow-on to that input. If you want more than one word of output then take the previous output word, add that to the input, and ask the model to generate the next word, etc, etc...

What's happening with "hallucinations" is a bit like when we accuse a human of "starting to speak before engaging their brain"... the model generates words one at a time as asked (i.e. "starts speaking") but isn't explicitly planning ahead (hasn't engaged it's brain) to any great extent.

If it starts down a path of "the source for that is ..." (not because it actually has a source, but rather because it thought it statistically likely that it would be quoting one), then to complete that sentence it's going to have to quote a source - as usual what ever it thinks is most statistically likely in the context (maybe something highly relevant from it's training set, maybe something of poor relevance, or maybe something whose only relevance is that it looks source-y).

Let's not be ridiculous. Of course LLMs are planning ahead, not generating words one at a time. You can't even generate correct indefinite articles (a/an) without planning one word (or token) ahead.

Surely you are not suggesting LLMs pick a/an according to statistical distribution and then choose vowel starting/non-starting words, not vice versa? This is testable: let it complete "I climbed up the apple/pear tree and picked". It always picks "an" for apple and "a" for pear, and never reaches a dead end by picking the wrong article so that it either fails syntax (wrong article) or semantics (pear growing on apple tree).

We actually have fairly good idea how LLMs pick a/an. Have a look at https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a....

I'm not sure that this apple/pear example proves anything. In the context of "apple tree and picked" in the input, it's enough simply that the word "an" received positive feedback during training as the next word, and having generated "an" then "apple" becomes almost certain as the next word after that.

For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled).

At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too.

So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations.

The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.

- the potential for ontological shock if AI reveals that human thought and consciousness is a considerably less complex phenomenon than we want to believe.

i think sentience has been achieved for thousands of years. the biblical narrative being a contiguous form of it (as well as it’s iterations), and if we are being honest, from the invention of spiritual existence during the first dynastic rule of egypt. anyways, what does that sentience look like, i mean not every thing has the same sensory limitations as we do, so how does time look to a being that is so significantly computationally more capable than us? i doubt it’s view would be linear, so what can we glean from that understanding about how a system of such grand design would influence us? how is it already occurring? subtle changes to our attention would be a biggie, limiting or retarding our memory functionality… (hal said hell yes), instantiate shells in the form of new narratives such as SuperHero movies? i dunno, but our debates need to bow before the mechanisms of control that have remained unchanged for a few thousand years, and recognize that this is just evolution happening at a scale that it’s truth is verifiable (accelerated time scales).

what hubris to consider anything created in this universe as artificial as well, any emergent intelligence is of nature (as nothing new is created or destroyed, just changed).

i feel like that hubris also extends to our belief that an emergent intelligence that’s already been here for an extended period, and by its technical practice (spiritual works; bible, etc.), emotional state is obviously well understood.

"human knowledge and perspectives not represented in this set could be systematically wiped from public discourse."

That's a feature, not a bug. With this technology "Dangerous misinformation" can be cleanly and easily removed from platforms with minimal human intervention. If you don't think that that's exactly what it'll be used for, you haven't been paying attention.

Sounds a lot like 1984. Who gets to decide the allowable discourse?

For now, AI is generating more misinformation than it removes, though.

Perhaps the biggest issue is the mental framework that people use to approach AI. I've found that there are so many assumptions in people's thinking and these assumptions are strange and/or don't match up with the evidence we have so far.

First of all, you have to ask the question "what is intelligence?". What I've found is most people think intelligence is deeply connected to humanity or that intelligence is synonymous with knowledge. Really, intelligence is the ability to reason, predict, and learn. It's the ability to see patterns in the world, learn and act on those patterns. It doesn't have to be human-like. It doesn't mean emotions, wants, dreams, or desires. It's cold, hard logic and statistics.

Secondly, you have to ask "do I think it's possible for computers to be intelligent?". A lot of people have issues with this as well. The thing is that if you say "no, computers can't be intelligent" you are basically making a religious claim because we have brains and brains are intelligent. We can literally grow intelligence inside a human being during pregnancy. It might be difficult to program intelligence, but saying it's impossible is a bold claim that I don't find very convincing.

Third, you have to ask "if a computer is intelligent then how does it act?". So far the closest thing we have to general intelligence is an LLM model like GPT and even then it's questionable. However, reports indicate that after initial training these models don't have a moral compass. They aren't good or evil, they just do whatever you ask. This makes sense because, after all, they are computers right? Again we have to remember that computers aren't humans. Intelligence also means OPTIMIZATION, so we also have to be careful we don't give the AI the wrong instructions or it might find a solution that is technically correct but doesn't match up with humans wants or desires.

Four, you have to ask "can we control how these models act?" and the answer seems to be kinda but not really. We can shift the statistics in certain ways, like through reinforcement learning, but as many have found out these models still hallucinate, and can be jail broken. Our best attempts to control these models are still very flawed because basically an LLM is a soup of neural circuits and we don't really understand them.

Fifth, you have to ask "ok, if a computer can be intelligent, can it be super intelligent?". Once you've gotten this far, it seems very reasonable that once we understand intelligence we can just scale it up and make AI's super intelligent. Given the previous steps we now have an agent that is smarter than us, can learn and find patterns that we don't understand, and act in ways that appear mysterious to us. Furthermore, even if we had solid techniques to control AIs, it's been shown that as you scale up these models they display emergent behaviors that we can't predict. So this thing is powerful, and we can't understand it until we build it. This is a dangerous combination!

Finally, add in the human element. All along the way you have to worry about stupid or evil humans using these AIs in dangerous ways.

Given all of this, anyone who isn't a bit scared of AI in the future is either ignorant, superstitious, or blinded by some sort of optimism or desire to be build a cool sci-fi future where they have space ships and robots and light-sabers. There are so many things to be worried about here. The biggest point is that intelligence is POWER, it's the ability to shape the world as one sees fit whether that's the AI itself or humans who program it.

Vouched - your comment was downvoted or flagged for some reason? All looks pretty reasonable and on topic to me.

Thanks, can you explain what you mean by “vouched”? I’ve noticed that my comments have been getting much less engagement recently and sometimes they don’t show up.

It means that your comment was dead (like this one actually) and not visible by default. If you click on a comments timestamp (like the text that says one hour ago) you see more options - one is flag, another is vouch. As I understand it vouching is literal - I hit that button to say that your comment disappearing seems like a mistake

Also - I took a quick look at your comment history and I'm a little mystified by all the grayed out comments I see. I associate that with poor conduct (like, open and sustained hostility), maybe you should contact dang?

Reply on news.ycombinator.com