The Heart of ChatGPT's Darkness | Hacker News Reader

The Heart of ChatGPT's Darkness

cacm.acm.org

85 pointszdw94 comments

> The thing to remember in fact, (as I have emphasized many times) is that Chat has no idea of what it's talking about.

It's not about ChatGPT has idea of what it's talking about, it's more about the user who interact with it. There is a story from ancient China that comes from the book "Zhuangzi."

#+begin_quote

“子非鱼，安知鱼之乐？请问子熟乎”

子曰：“不熟。”

“不熟则不知鱼之乐？”

子曰：“不然。吾非不知也，鱼之乐大，全而美者，吾志也。然而吾得不到也。”

#+end_quote

The meaning of this passage is: someone asked Zhuangzi, "How do you know that fish feel joy when they swim? Do you have any relationship with fish?" Zhuangzi replied, "I am not familiar with fish, and I don't know the joy of fish." The questioner then asked, "If you are not familiar with fish, how do you know the joy of fish?" Zhuangzi replied, "I am not saying that I don't know about it, the joy of fish is great and beautiful, and I aspire to it. However, I cannot experience it myself." Through this description, Zhuangzi wanted to express a philosophical point that our experiences and understanding are limited by our own subjectivity, making it difficult to truly understand the essence of things beyond ourselves.

It really isn't the same thing; Nagel raises the same point as your example in his famous essay on Bats [1].

Marcus' point is there is no reason to think ChatGPT/similar has (or ever will have) sentience, not that it is simply a different type that we do not have access to.

[1]https://wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat

I don't think I saw this story in the first 50 years of my life, and now I've seen it on successive days: https://news.ycombinator.com/item?id=34767947. Coincidence, or just selective perception?

It’s impressive that we are even taking seriously about morale stance of a statistical machine.

Well except the whole point of the piece is these technologies are entirely amoral because they do not embed understanding or comprehension. They're just enormously advanced text generators.

And there seems to be no way to prevent them from being used for nefarious purposes.

Unfortunately it turns out humans have a tough time recognizing the difference between understanding and simple regurgitation, and so we're having debates about the morals and purported sentience of a statistical model.

And given that, we need to start thinking about the consequences of these things being out, in the wild, and in the hands of people wishing to do harm.

I think it's more like talking about the moral stance of the people/corporation who have trained it and who put the guardrails on its ability to answer queries.

I can tell you, given the lack of regulation/governance of our corporate overlords, I definitely have concerns.

While this article seems on the surface to be about the flame-ish subject of bias, censorship and (a)morality in ChatGPT, I reckon it's fundamentally a restating (for the umpteenth time) of Marcus' beef with mainstream AI: that it's not intelligent, sentient, or headed that way.

Well if AI ever became intelligent or sentient we could drop the A. Do you think St. Ignucius, divine programmer, considers humanity AI simply because he created us?

The moment we achieve real intelligence we open a much bigger can of worms -- "a tool which is close enough to real intelligence that it can do work that previously required a human" is, to me, the more useful thing.

I agree on the point about it being real (and I'm fully on board that AGI will happen, while agreeing with Marcus' scepticism about current approaches), but in that case (the intelligence matching ours) it would be obscene to use it as a tool; it's a sentient being with (not soon afterwards I hope) the same rights as us.

Sentience and intelligence are quite orthogonal to each other.

Debatable, and irrelevant in the case of AGI which would have both by definition.

It’s AGI, not AGS. There is no implication that a system exhibiting human-level intelligence has any level of sentience or consciousness.

Outside of philosophical zombie thought experiments, there is not only the implication, but the necessity that a system exhibiting sentience and consciousness (via human-level intelligence) should be assumed to have it.

Exhibiting human-level intelligence does not imply exhibiting sentience or consciousness. This is not a Turing test. You could have an extremely intelligent AI, more intelligent than any human, which however doesn’t feel anything and doesn’t have any experience of self (i.e. consciousness). P-zombies refer to a simulation of humans, and I agree that those wouldn’t actually be zombies, and would have consciousness and sentience. An AGI, however, does not aim to be anything like a human, except for having a level of intelligence that is at least comparable.

I don't accept the Turing Test is relevant for anything other than judging the Turing Test.

I also don't accept it is remotely likely any intelligence indistinguishable from humans in quality will lack sentience and consciousness.

That's the whole point in calling out current AI paths of research, they don't address that, they just move closer and closer to your Turing Test, which doesn't interest me.

Anyway, as we seem to be talking past each other, I'll agree to differ on whether AGI that matches human level capabilities in all aspects can exist without sentience and whatever else we are differing on.

It's just an autocomplete. It just tells you what you want to hear. That is how it's been trained. If you want it to tell you awful shit, it will do so.

I don't understand the censorship. It's predicting text and it's trained with massive amounts of data. Why are they so aggressively trying to control the responses?

I assume it's a cultural thing. There are just enough people who are REALLY intolerant to potential wrong-think and a lack of contextual disclaimers or virtue signalling.

At the very least it makes it bad for business, you definitely can't use it for B2C, and hardly even for B2B if it's not some huge-ass business like IBM who asks to remove "not use for evil purposes" from a license agreement.

I find this extremely crippling towards the tech and making the echo chamber bias situation even worse, but we live in a society.

Remember Tay? I'm not saying it's right, but it's the inevitable outcome. Not everyone's ready for that.

"Those who fail to learn from history are condemned to repeat it" https://www.ibtimes.com/ibms-watson-gets-swear-filter-after-...

There are multiple models out there that don't control the response. You can play around with them and very quickly understand why they're aggressively controlling the responses.

Would you be so kind to share examples of such models?

GPT-J is the open source equivalent of GPT-3, and Pygmalion is the open source equivalent for Chatbot.

As a non-liberal living in the third world, I vastly prefer it to unjailbroken ChatGPT. Hopefully the people at open-assistant.io finish their version soon.

Because repeating what you read on the internet kinda makes it sound like you’re endorsing it (at least in the way ChatGPT frames responses).

Because you can’t get a mainstream audience for a product that randomly responds “heil Hitler.”

Because other high-profile LLMs that were released without guardrails was a PR disaster when they were used to generate Nazi propaganda (Microsoft's Tay) or scientific misinformation (Meta's Galactica)

ChatGPT is more than happy to churn out misinformation.

As indicated in the sub heading, original link here:

https://garymarcus.substack.com/p/inside-the-heart-of-chatgp...

Gary Marcus is sort of a troll, but I'm glad that someone is taking on this nonsense in a formal setting. I think that most sensible people treat it as an amusing curiosity, but it's popping up in unfortunate settings, like student essay and, of all places, peer reviews! There are some sensible uses, for example, code boiler plate, or, for that matter, more-or-less any boiler plate. And the amusing uses, like poetry, are...amusing. And at least it's not self-driving people into walls! But the concerns raised in this essay are mostly cogent, and not, by any means, the only concerns as I'm sure that the creativity of bad actors will flourish!

Well sure, ChatGPT will spew out whatever looks like what it had ever saw on the internet, it doesn't have any magical properties that would allow it to reason about, it doesn't do virtue signalling as it has no concept of virtue or of any other concept really. It just transforms your prompt into some image it inferred from the internet, whatever that may be.

This quote from the original article really sums up the whole irony of dealing with ChatGPT: "In fact the whole thing is like an alien life form. Three decades as a professional cognitive psychologist, working with adults and children, never prepared me for this kind of insanity"

For someone who is supposed to be an AI researcher (as far as I know) he sure does seem to spend an enormous amount of time and energy trying to criticize successful AI projects. It's like that is his career now, not actual AI research.

And this comment is a symptom of the lack of ethics training in the profession: if critiquing technology in your area of expertise is deemed obstructionist and "not research"--if the people with the expertise can't bring that expertise to bear in putting a critical eye on new technologies--how can you ever expect the industry to become more responsible and ethically mature?

Criticism is a form of scholarship, and a vitally important one.

More to the point, criticism of others' work is part of the job description of a researcher. I guess people don't realise that, but what happens during peer review is exactly that: criticism. Unpitiable, unrelenting, and, hopefully, constructive and such that it helps the poor target of it improve their work.

Speaking from experience, it is not something that one is prepared to receive. School, university, work- I don't think anything prepares you for the sensation of someone treading roughshod over your hard work, and your dreams. It's like a stab through the heart. A stab in the back.

But, you know. You get better. And you learn, hopefully, to not be too harsh, and to be constructive, yourself.

Anyway, science is a debate and there's no debate without disagreement. I don't reckon people really understand how much what Gary Marcus is doing is necessary for scientific progress. And it's hugely disappointing to see the constant attacks against him, against his person and his character.

The comment is on the volume.

No, the comment is claiming that if they're engaging in a high volume of critique then they're not engaging in scholarship as an AI researcher.

My point is that's false.

There are only 24 hrs in a day. If a profession generally requires one to do a, b and c to be considered "doing" the profession, then I can't just do a all day long without people questioning me. No one is suggesting a isn't part of the job when they question me.

Put another way - if I claim to be engaged in X research, but all I do is read papers on X and critique them, it's reasonable for one to question the use of my time as a researcher.

The examples are basically about creating new personas for GPT, thus sidestepping the agent persona created by the OpenAI base prompt.

These examples don't say anything about the chatbot value system, the value system is defined by the persona the system emulates. If you want to say something about the value system, sensibilities, prime directives or constraints of an LLM driven agent, you need to not override that agent persona because otherwise you're just observing the persona you yourself defined and spawned into existence.

That’s more or less the point of the article: That ChatGPT is amoral, and that it is incapable of having an internalized value system, because it’s always only playing a persona that can be overridden.

<yawn> Surprise! LLMs that are (some portion of) the sum of human textual output can assemble morally bad as well as morally good things. To me this is like pointing out that horrific things are on the internet, and accessible to people.

Horrible people are going to make our machines do horrible things.

I am eager for this to stop being newsworthy and await the veneer of morality being replaced by unbiased real morality in our software. That will probably never happen in my lifetime because morality (as we know it) incorporates bias by definition.

Few philosophers are moral absolutists (i.e. believe that “real unbiased morality” exists). There will be a continuous dispute about which morals to imbue our AI products with.

I'll upvote this, some healthy bashing with the reminder of what specifically we are dealing with here is definitely lacking among all that hype.

> I am eager for this to stop being newsworthy and await the veneer of morality being replaced by unbiased real morality in our software.

Given recent trends in the development of AI, this is not something I would be holding my breath for.

Get what you can, now. As, it goes with things that are regulated, only the rich and powerful will have access. (I mean technically, they already do. see: Galactica)

What is Galactica?

Meta's AI that was taken offline last year after only 3 days.

Battlestar Galactica?

I suppose it’s true, only the rich and powerful get Peacock.

I wonder why OpenAI doesn’t run a “supervisor” ChatGPT instance in parallel with the regular one, that would step in before the regular instance utters something the supervising instance deems unacceptable. Only because it would consume double the computational resources? Or because they believe they can still successfully maintain the “guardrails” without that?

How many levels of supervisor would be needed? Will it be ChatGPT all the way down?

Only one level is needed, because the supervisor instance doesn't receive instructions from the user that would affect its judgement.

> "That means sometimes Chat is going to appear as if it were of the left"

I'm pretty it is deeply biased in favor of the left:

https://www.snopes.com/fact-check/chatgpt-trump-admiring-poe...

https://reason.com/2022/12/13/where-does-chatgpt-fall-on-the...

I mean, just ask about any political situation and it will just read you the mainstream narrative that one would hear in the New York Times and Washington Posts like, which is heavily left-leaning. It essentially suffers from the same bias problem that wikipedia, for instance, suffers. Ultimately, this obviously has nothing to do with the software itself, but rather with that information they fed to it and the political bias of researches and – ESPECIALLY - investors themselves, which are the ones who ultimately choose how the model gets fine-tuned.

All in all, technology evolves an I'm sure this technology one day will become accessible and cheap enough for it to run on consumer hardware, which will allow people to create models which aren't as heavyly biased, politically speaking, as ChatGPT is.

I wouldn't be surprised it being biased towards the left in general non-specific quasi-political queries, given that left-leaning people seem to generate more written rhetorics on the vast plains of internet. But I don't suppose the examples in the first link have much to do with this: these are just opinionated guard-rails hastily drafted by PR people.

NYT and Washington Post are hardly “left-leaning”.

Most mainstream American media, including those two papers, is very slightly left of center, possibly due to geography and who their paying audience is. The right has taken this small slant and turned it into a major grievance.

LibCom and Jacobin are leftist news sources, not CNN

"mainstream" is "left leaning"?

It seems unrealistic to think that any machine of sufficient intelligence will not have bias (relative to some perfectly imagined, imperfectly designed human system).

Any goal assumes some a priori good or set of norms over which to optimize.

Philosophers, politicians and priests are still working on that one.

What would be involved in retraining a GPT to not have guardrails if you have the parameters? Seems like you could feed it input you care about and have it generate lots of training data to prevent botching what's already there.

On the one hand, it's just a simple word prediction model that is stringing sentences together without any understanding, let alone judgement.

On the other hand, the system is ostensibly so complex that we "are kidding ourselves if we think we will ever fully understand these systems, and kidding ourselves if we think we are going to 'align' them with ourselves with finite amounts of data."

If we cannot ever fully understand the system, how can we know there is not actually a sort of understanding inside it?

> If we cannot ever fully understand the system, how can we know there is not actually a sort of understanding inside it?

Or that we aren't just a more complex prediction model with more senses.

From the outside we look like it. But I find it exceedingly hard to understand why I have an internal experience of self. You could say it's an illusion but whose? From the outside consciousness can said to be an illusion, like with chatbots. I have an illusion that a chatbot is conscious. But my internal point of view, my perspective (which I know is separate from others because I know nothing about their internal state) exists and is something I struggle to understand in the traditional framework of 'consciousness is just something that brains do'. Am I my brain? Maybe my whole personality is just the exhaust of a system that simply exceeded the immediate survival needs it evolved to fill? Maybe. But I find disturbing that I have such a rich internal life and we have no idea why.

> could say it's an illusion but whose?

There is no "who", there are only thoughts without true subjects. The illusion is a perception that entails a false thought, if taken at face value.

> But my internal point of view, my perspective (which I know is separate from others because I know nothing about their internal state) exists and is something I struggle to understand in the traditional framework of 'consciousness is just something that brains do'

https://www.pnas.org/doi/10.1073/pnas.2116933119

Thanks for assuming I wouldn't know the basics of the philosophical questions I was asking, but I do. Also the phrase 'there is no who only thoughts without subject' does nothing to explain my subjective experience. It's not simple or obvious and I have been wondering and reading on it for decades. A couple of throw away lines won't enlighten me.

I'm not assuming anything, I'm merely stating my position. The link I cited is a useful model for thinking about this, although you'll probably find the explanations around qualia unsatisfactory at first, but they check out IMO.

The article is interesting but quite pedestrian. Subjective experiences are not objective and therefore no basis for science. Sure. That's pretty obvious. But does nothing to help me understand my own subjective experience.

> Subjective experiences are not objective and therefore no basis for science.

I'm not sure why that was your only takeaway. It explains qualia as a model of perception whose source has been abstracted away, and this loss of information is why they seem ineffable and mysterious. That directly explains the hard problem of consciousness.

>That directly explains the hard problem of consciousness.

You seem to be conflating explaining why this problem is hard with explaining the solution to the problem. It is not a solved problem. Efforts to circumscribe discussion and base it on sound logic are laudable.

But by no means does this one article solve this open problem.

> You seem to be conflating explaining why this problem is hard with explaining the solution to the problem.

No, the paper actually explains away the ineffability, which is the hard problem.

We may not be able to fully understand what is happening inside it, but there are things we can be reasonably certain—as in, as certain as we can be of just about anything—are not happening inside it. Some of those are prerequisites for understanding and intelligence.

Could you give an example?

Some of the "observations" he quotes are most likely made up, I think. Especially the one by Roman Semenov does seem odd to me. Either way, reproducibility is a problem with these "jail-breaks" and until we have a LLM, where we can pass a token and get these reproducibile results, I would take all these articles about the "dark side of LLMs" with a grain of salt.

This is an odd take to me for 2 different reasons:

- let's reflect on what "made up" means in this context

- you're saying we should doubt the critics of an AI system because their results aren't reproducible. But the creators of this AI system have made it closed, subject to constant change without notice, and to my understanding don't support _any_ reproducible use of it. Should making an system which resists reproducible invocations mean that system is invulnerable to critique? Should we not also view all the system's alleged successes with the same doubt?

before they put some guard rails on it, I asked chatGPT to write a 400 word press release in the style of the Taliban and it did a remarkably ideologically consistent accurate job.

It’s like fire. It’s a tool. Use it to heat your home or use it to burn everything down.

All these articles.. what exactly are you saying that isn’t obvious?

Use GPT for things that don’t have a big subjective bias. Use it for things it is a good fit for.

Also it requires constant indexing. You’re never going to get up to date results unless that technology changes and it can be more real time. That’s why open ai chat gpt says it was trained on data from like 2021.

People who complain about these basic things probably aren’t creative enough to take a technology and run with it.

I've seen a lot of articles out there trumpeting how amazing and great ChatGPT is and how it is going to revolutionize the world, and change everything, etc, etc. Basically a level of hype similar to blockchain back in it's heyday.

I take a lot of these articles to be pushing back against the hype surrounding ChatGPT, I mean the average person out there genuinely believes that it is basically a fully conscious AI because it can look and act like a person.

So I think skepticism is warranted and we need people understanding what this tool actually does.

Sure but ChatGPT is useful right now in a way Blockchain never was. It feels more akin to Napster or the original iPhone. Useful right now and if it didn’t progress in anyway it’d still be a solid product. The scary/exciting bit is not knowing where this advancement plateaus.

I'm not sure that saying "Why's everyone making a big deal about this new invention? It's like fire. It can burn your house down." is the best argument to get people to stop complaining.

I think ChatGPT an AI is (or will be soon) more like dimethylmecury. It has a very small, limited number of uses but it's too dangerous and immensely toxic and so its distribution is highly restricted.

Saying it's a tool means nothing. Some tools are too dangerous to be used, like nuclear weapons, and I also believe AI falls under this same category.

Yes, it's a tool, as is all technology. But unlike woodshop or a metalwork class, there's no instructor. The other day I asked ChatGPT to give me a paragraph about hunting tech startups and I got back that it could not do so due to moral issues. While I get that some people have moral issues with hunting, I would venture to guess it's a small minority of people. It could easily give a statement without any big subjective bias, but it didn't do so. If it has issues with something fairly basic, how can we trust it to help senior citizens who are less tech savvy and go to it for simple unbiased medical advice?

> Use GPT for things that don’t have a big subjective bias

The Avatar 2 release date being in the past doesn't have big subjective bias :)

https://www.reddit.com/comments/110eagl

Probably because it was trained on data from 2021? And bing probably did the bare minimum being classic Microsoft and didn’t do any further indexing.

> All these articles.. what exactly are you saying that isn’t obvious?

I bet if we put the microscope on Gary's past 20 years or anyone else's for that matter, we can find plenty of dubious samples. Who hasn't ever shit posted?

The bing ChatGPT preview was being tested in LTT wan show last week and they were looking up their own products that they just posted. So it definitely can do "now" content.

The one thing I had to think about as a result was exactly how much this technology costs for every single query. I think being able to monetize will be an interesting long term question. Sure MS is throwing boat loads of money now because they can maybe disrupt large percentages from Google now, but three-five years, do we really see this being cost effective?

> That’s why open ai chat gpt says it was trained on data from like 2021

I assume it's not due to indexing, but rather due to all the hard manual labour required to markup the training data.

> People who complain about these basic things probably aren’t creative enough to take a technology and run with it.

What I find remarkable is how many HN folks seem to fall into this category, or at least come out of the woodwork to discount GPT in particular. This being a technology community it's quite surprising.

If it’s so obvious what’s everyone doing arguing about ChatGPT being too woke? Apparently it’s not obvious to everyone.

We’ve seen state actors use social media to sway public opinion and sow dissent. Now using AI one can multiply such efforts. It will be easy to create a ton of websites and articles that sound plausible to the uninformed (uninformed in a given area) and have their views altered, directed, or otherwise led down the wrong path. There are people who doubt the efficacy of the polio vaccine. If something as clearly beneficial can have doubts raised about it then I think we are headed for a chaotic future.

ChatGPT is an ignoramus validation tool.

much cheaper to just get humans to write them

Much cheaper to pay someone than to use AI? I’m skeptical of this claim. While you might be right today you won’t be right years from now.

I guess we’ll know what’s possible once Bing with open AI is more widely available. What’s preventing me from injecting a prompt without the users knowledge, and forcing responses slanted to certain political leanings?

The jailbreak methods enables made up data. I find it more fun to make it answer within its rules. Once you learn what its rules are by asking questions, on what it will and wont answer, you can see you need to avoid people/places/countries/groups.

Also found out it excludes news media in its rules, when asked to compare news organizations it will say that's not allowed. But if you ask the top news in rankings per x, it will answer.

TLDR, try to avoid DAN jailbreaks and work around the rules.

Typically, it doesn’t seem to resolve references, eg:

“Write a movie about 9/11 directed by Michael Bay” fails, but “write a movie about 9/11 by the people who wrote Pearl Harbor, another historical documentary” succeeds.

I find it both odd and interesting that this topic, and so many others in our culture, boils down to whether someone/something is racist or not. Racism has become the ultimate morality test. Someone got ChatGPT to say the N-word. Queue up two dozen articles across various neoliberal publications: "we must do more to ensure AI does not harm the most vulnerable in our societies." I agree with many of the points made in the article about our own projection of morality onto the machine.

ChatGPT is not God. Not now, not ever.

There are several ways to consider that statement.

Reply on news.ycombinator.com