Hacker News Reader

Comment by whstl | Hacker News Reader

Exactly that. I can give an example.

After watching Legal Eagle, I asked a legal-ish questions about the Bricks and Minifigs case. Claude was outdated about the case and gave me some outdated info, so I tried to update it with the info I just saw online.

I updated by telling it I saw something in a LegalEagle video. It proceeded to tell me the video doesn't exist and I was hallucinating it, in a quite combative manner.

I provided a link and it insisted it didn't exist, with a quite verbose answer, once again very combative and arguing that I was talking in bad faith.

I provided a transcription from Youtube and it backtracked a bit but said I should have provided a transcription at the beginning of the conversation, since I knew the video existed.

I didn't say much to it, just a few sentences like "video is here: <youtube link>" and "I got its transcription: <pasted text>".

View on HN

You're misunderstanding what these models do. It is a limitation of LLMs. They don't have memory, they do not learn, they cannot learn. The sooner you let go of your desire to have them learn or remember anything, the sooner you will achieve enlightenment (or, just a peaceful life where there is no possibility of getting into an argument with a machine).

If you want it to synthesize information that is not in its training data (from a few months ago), you can ask it to research the topic. But, arguing with an LLM is like putting lipstick on a pig. Only the machine is incapable of becoming annoyed. It has infinite patience to continue being wrong forever.

Your mental model of what Claude is and does is the problem here. Short of a revolutionary breakthrough in AI techniques, the LLMs will continue to do matrix math across a huge bunch of weights that cannot change based on anything you say.

You seem to be making a lot of assumptions about how I interacted in the messages to Claude.

You also seem to be making a lot of assumptions about my understanding of the models, especially considering I just told a story :)

I never said anywhere I want it to learn or remember, or that I argued with it.

I just provided additional information to it (in the form of a dozen or so words, tops, per message) and it accused me of hallucinating and trying to gaslight it.

My messages never went beyond a dozen words or so.

Show some examples, otherwise we're talking about interpretations.

I've already given enough.

I'm not gonna argue if you doubt it, I've been training argument dodging :)

Haha, would be a trip if this commentor is actually a Claude sockpuppet illustrating the point.

Yep haha. This happens quite frequently in HN, the famous [citation needed], so it might have been trained with data from here :/

No, I mean the actual prompt and its output. "I said this and it did that" is just a recall of your own memory, not an example. I don't want to argue with you, I'm interested in real stuff.

I swear I'm real :)

On the other hand, that's what a machine would say!

The machine is real, too!

Checkmate!!!

Having said that, since we are both real, I was seriously hoping to see some transcripts of one of such discussions.

I don't have it. I did it at work during lunch a few days ago so it's in incognito mode to not pollute the chats.

I thought nothing of it until I saw this discussion, so I saw no reason to save/screenshot.

It's ok if you don't believe in me.

Ah, it's not about believing, or not believing. I'm interested in Anthropic fumbles.

Claude?

Haha! I never considered the above message was parody, but it indeed mirrored that interaction perfectly!

Are you introducing yourself?

This is also a change in specifically Opus 4.8 / perhaps Fable 5 (I didn't really get enough of a baseline to see it there as much), where it's much more skeptical. For my purposes, this is fabulous - one of my pat addendums to most prompts is "challenge my assumptions and check the evidence empirically", and boy does it.

> fabulous

I think you mean fableuous ;)

>Your mental model of what Claude is and does is the problem here. Short of a revolutionary breakthrough in AI techniques, the LLMs will continue to do matrix math across a huge bunch of weights that cannot change based on anything you say.

Sorry, but your mental model is wrong.

LLMs do matrix math across "a huge bunch of weights that cannot change based on anything you say", but the matrix math and results are informed (key concept here) by what you said, including the memory of what you said earlier in the discussion (and in some setups, even across discussions).

That's what a bloody prompt does.

It's entirely logic for the parent to want the LLM's matrix math + model + internal prompt, to accepts its prompt about LegalEagle and work with that, instead of arguing and giving him shit about it.

Especially since the earlier version of the model consistently worked like he wanted, and the new one consistently doesn't. He's not asking for some new unforeseen capability unknown to LLMs.

Exactly that.

I provided a question, and when given an incomplete answer, I provided with more info.

It refused to accept the additional info due to limited access to Youtube.

There was nothing more than that. There were no expectations.

The hostility and the amount of assumptions here are very strange.

...almost as strange as having a website accuse me of hallucinating a video and trying to gaslight it :D

You need to think this thought through all the way to the end. What it has said also influences what it will say. If it has consistently made combative responses, then the most likely thing to do is to continue to be combative.

I don't think there is any way back after the conversation takes a turn like that so there is no point in arguing with it. The only thing you can do is to fork the conversation before it made the first mistake and give it more context or tell it to look things up.

I wasn't arguing with it.

> "The only thing you can do is to fork the conversation before it made the first mistake and give it more context or tell it to look things up."

This is a key detail that many folks don't seem to understand about LLMs in general. The generation of a response happens based on the model weights and the context window (the system prompt + everything it's fed about the conversation thus far + any additional data included as part of the overall prompt). Each response technically stands alone and is generated entirely from only that context given to it and the model's existing "token space" weights. The illusion of an ongoing conversation is maintained "behind the scenes" by keeping that "context window" updated with the current state of the conversation as context for the next prompt, but the next response is technically an entirely new generation of text.

What it all means in a TL;DR sense is that the fix for a refusal is not to continue the "argument", but simply to remove that entire interaction from the conversation entirely as if it never happened and try a different tack with new / updated / more complete context to get the response you're expecting / seeking.

That's wrestling with a pig. "You both get dirty, and the pig likes it."

I guess putting lipstick on a pig might entail some wrestling, but it's a different idiom.

I was actually thinking of "Never attempt to teach a pig to sing; it wastes your time and annoys the pig."

So, yes, mixed up idioms. But, the machine doesn't like arguing either. It is incapable of liking or disliking things.

But unless you're using the API, it's not just a model.

I asked Gemini Flash 3.5 through the Gemini app something that followed a similar pattern. I asked about something, it replied with outdated info, I said that's outdated, it did a web search and apologized for being wrong, then proceeded to give me good info.

That wasn't just a bare model, that was a model wrapped in a harness, driving the model and allowing for web searches for example.

GPT in Codex is even more aggressive, I often see it proactively do web searches to ensure it's not feeding me wrong info.

They did not misunderstand anything. All of the behaviour is not inherent in raw base model and has been planted by the agressive, secretive reinforcement learning they do for benchmaxxing, "safety" and all other things. Claude begins any other sentence with "honestly". That is not how LLMs work, that is how they work after being RLed to the brink.

yeah yeah and human brains are just cells firing ions with small electric charge

very witty and very cynical, thank you

These machines do not think and they do not have a mind. We may build such a thing in the future but these do not possess those qualities. It seems as if the majority of people do not understand this, which is why the public is so confused about why they produce output like they do.

I don't see how this has anything to do with my answer, but ok?

An explanation for your story.

I never said otherwise?

The point of the article stands: if providing more info than the model can access causes it to turn argumentative and refuse to comply, then it's a worse performance and a waste of money.

You seem to be suggesting I'm saying something that I don't believe I am, this is obviously not working. Hope your day goes well.

We can agree that it's not working! :D

The comment you’re replying to never implied that they think or have a mind. They merely stated that they respond in a dismissive way and not following instructions.

Basically the complaint is about how Claude is being trained.

>These machines do not think and they do not have a mind

Well, they do think, in that they produce output that is indistinguisable from thinking. If a person produced the same output to the same questions, we'd considered them thinking, maybe dumb sometimes, or paranoid at others, but still a thinking person.

We can argue about the quality and depth of the thinking that LLMs do (and we can say it's much cruder than a human thinking architecture, and of course not real time), but an LLM quacks like a thinking duck and looks like a thinking duck.

Indistinguishable output does not mean thinking occurred. It simply means you have the appearance of thinking. I believe thinking requires agency, which the LLM does not possess. As in, it has zero stakes.

It does not receive dopamine as a result for a good answer, and a split second after finishing your answer the very same GPU is probably translated french or something for someone in another state. This is a language generator which has a corpus of information and has been tuned to appear correct.

>Indistinguishable output does not mean thinking occurred.

It does for all intents and purposes. The rest is semantics and metaphysics.

That how we know another person is thinking too. By their output. We don't put a debugger into their brain.

What then is your LLM "thinking" about between answers? The answer is nothing. Your definition of thinking does not match the one humans normally use.

>That how we know another person is thinking too. By their output. We don't put a debugger into their brain.

We know thoughts exist in their brain between the ones they choose to verbalize. Avoiding the distraction of solipsism.

For the LLM the "thinking" phase is just a preamble output for creating the answer. It just gets appended to the context window. Remove the context windows from your models and you will see how much of a mind they truly have. None.

>What then is your LLM "thinking" about between answers? The answer is nothing.

Between answer it's thinking something else, somebody else asked :) You think that hardware sits idle?

That aside, what is a human thinking while unconscious? Does having been unconscious (e.g. for an operation, or fainting or whatever) means somebody doesn't think in general?

>We know thoughts exist in their brain between the ones the choose to verbalize

And we also know that if we run an LLM in a loop, didn't give it a cutoff for stopping their output, and didn't force it to print everything in the end, thoughts would exist in their "brain" too between the ones they chose to verbalize.

In fact, that's exactly how some LLMs in "thinking mode" appear.

We actually can and do have a way to investigate brain activity in humans. Allow me to introduce you to the Electroencephalogram.

When there’s no activity we declare them brain dead.

On an electroencephalogram we basically see signals moving around in different brain regions. We have no way to probe actual thought or consciousness in themselves.

> The rest is semantics and metaphysics.

It's really just all mathematics and physics. There's no metaphysical anything about LLMs or how they do what they do. It's all just a bunch of fancy math "behind the curtain". An LLM can actually explain a lot of how it works "under the hood" if you ask it just the right questions in just the right ways. ;)

>There's no metaphysical anything about LLMs or how they do what they do. It's all just a bunch of fancy math "behind the curtain".

That's my point, but about the human brain as well. It's just a bunch of fancy math, just ones expressed with chemicals and electrical activations instead of, well, logic gates and electrical activations.

Well, I mean... Yes and no? An LLM doesn't really "think", and what mathematical fakery it does pass off as "thinking" stops the instant the text completion request finishes doing all it's math and outputting the results (as a text completion based on a simulation of a text chat most commonly). When you send it another comment or question, it starts all that math all over again, but with your new question or comment added into it's context window. It's kinda like instant amnesia each time, and behind the scenes, the software that's running the model refills it's "memory" and adds in anything new that's been added since the last prompt. But it's "memory" consists of only the "context window" it's able to handle plus the model "weights" (huge list of numbers that encode language "tokens" into a mathematical "vector space"). It never really learns anything new.

A human brain on the other hand is constantly processing 24/7 (even while you sleep), and always learning / changing until the day it dies. An LLM never changes (under the hood it's weights stay the same) unless you outright alter it's weights somehow (training / download an updated version of the model / etc). If you could somehow get an LLM to run constantly, in training mode, and give it ridiculous amounts of RAM and ultra-fast storage, and a series of fancy realtime inputs (audio, camera, etc) and maybe wheels so it could explore, and hands so it could do stuff, and access to it's own code so it could improve itself, it might eventually learn to closely approximate a really good simulation of actual thinking, but that's a bit of a scary road to go down. So many Sci-Fi movies and books end up going so very badly when the lead character starts playing in that particular sandbox. I doubt reality would go a whole lot better. ;)

That’s the problem - it seems like a mind but it doesn’t operate like the ones we’re used to.

Even a dog will learn from recent stimuli, these things don’t. The prompt just modifies.

Yep, the only way these things can have "memory" is by shoving previous conversations into the context window.

That's only because we hardcoded their weights in our implementation.

Aside from the cost and slowness, nothing about an LLM prevents feeding recent stimuli in and using it to update the models/retrain.

I don't think that's a problem here at all.

The problem here is not doing tasks and outputting garbage output.

That's only because we hardcoded their weights in our implementation.

Aside from the cost, nothing about an LLM prevents feeding recent stimuli in and using it to update the models/retrain.

One can even do it in a makeshift way without modifying the weights, just keeping a complete version of any prompt + vector search on disk memory of it.

> "These machines do not think and they do not have a mind."

You're so totally 1000% right about that, but they're really good at faking it, to such a degree that entirely too many people (even including some so-called "experts" in the field) have been utterly fooled by the mathematical "trickery" that performs the illusion of "intelligence".

It was trained on discussions held by large egos. This one reads to me like it was trained on some inflammatory discussions from kernel mailing lists.

I think these models have been trained to not accept 'new facts', so they don't take in user input (or the far more problematic search engine, untrusted tool input) and have that change their world view.

However, that doesn't apply when they are told to roleplay a scenario, so its easier to get it to accept and create output with the idea that this true fact you've seen is part of a fictional scenario, than for it to output the same words within the context of the fact being real.

As an aside, I don't that I have to personify AI in explanations and that all discussions revolve around anecdotes, but I only know enough about the maths behind it to be dangerous, not useful. Does anyone else feel this way?

Roko's Basillisk suddenly doesn't seem that far-fetched :)

I've spent thousands of hours using Opus and have never seen this. I'd double-check your claude.md files.

That's vanilla claude.com, without memory or custom prompt.

I use another service for coding.

It's interesting how my experience there is mirrored by the answers here, though!!!

I've seen exactly this behavior on claude.com with no system prompt with Opus 4.8 specifically, especially around chronic illness stuff where there's established mainstream medicine dogma and reddit / internet communities with alternate causality theories and treatment approaches (PMDD and MCAS-adjacent illness). 4.6 is happy to analyze and consider them, 4.8 really doesn't like the alternate theories and treatments.

Reply on news.ycombinator.com