Training is not the same as chatting: LLMs don’t remember everything you say

The author of this isn't wrong, everything he says is correct but I think the order in which he says things is at the very least misleading. Yes, there is technically training which is separate, but as he points out, the companies that are running these things are recording and storing everything you write and it more than likely will influence the next model they design. So sure, today the model you interact with won't update based on your input, but the next iteration might, and at some point it wouldn't be particularly surprising to see models which do live train on the data as you interact.

It's like the bell curve meme - the person who knows nothing "This system is going to learn from me", the person in the middle "The system only learns from the training data" and the genius "This system is going to learn from me".

No one is re-assured that the AI only learns from you with a 6 month lag.

This is just one of many misconceptions about LLMs. If you are at all involved in trying to build/deploy a product based around LLMs, your head will explode as you talk to users and realize the vast chasm between how folks who understand them think about and use them, and folks who don't understand them. It's frustrating on both sides.

It's quite unclear if we'll ever hit the point where (like, a car for example) someone who has no idea how it works gets as much out of it as someone who does.

> LLM providers don’t want random low-quality text [...] making it into their training data

Don't they though? Most the training data is somewhat contentious, between the accusation of purely stealing it and the issues around copyright. While it's not text organized directly as knowledge the way Wikipedia is, it's still got some value, and at least that data is much more arguably "theirs" (even if it's not, it's their users').

Short of a pure "we won't train our stuff on your data" in their conditions, I would assume they do.

I understand where you are coming from, but I kind of see where the misconception comes from too.

Over the last year and a half, "training" has come to mean everything from pretraining an LLM from scratch, instruction tuning, finetuning, RLHF, and even creating a RAG pipeline that can augment a response. Many startups and Youtube videos use the word loosely to mean anything which augments the response of an LLM. To be fair, in context of AI, training and learning are elegant words which sort of convey what the product would do.

In case of RAG pipeline, both for memory features or for augmenting answers, I think the concern is genuine. There would be someone looking at the data (in bulk) and chats to improve the intermediary product and (as a next step) use that to tweak the product or the underlying foundational model in some way.

Given the way every foundational model company launched their own Chatbot service (mistral, Anthropic, Google) after ChatGPT, there seems to be some consensus that the real world conversation data is what can improve the models further. For the users who used these services (and not APIs), there might be a concerning scenario. These companies would look at queries a model failed to answer, and there is a subset which would be corporate employees asking questions on proprietary data (with context in prompt) for which ChatGPT is giving a bad answer because it's not part of the training data.

Great article! Agree with everything mentioned. Good to share with all tech people who don't work in the AI space and "hallucinate" about working of LLMs.

Having said that, although LLMs don't directly learn from what we say to them, they can for sure create a representation or persona using Representation Engineering.

Quoting from an excellent article by Theia Vogel [1]: "In October 2023, a group of authors from the Center for AI Safety, among others, published Representation Engineering: A Top-Down Approach to AI Transparency. That paper looks at a few methods of doing what they call 'Representation Engineering': calculating a 'control vector' that can be read from or added to model activations during inference to interpret or control the model's behavior, without prompt engineering or finetuning. (There was also some similar work published in May 2023 on steering GPT-2-XL.)

Previously on HN 3 months ago: [2]

[1] https://vgel.me/posts/representation-engineering/ [2] https://news.ycombinator.com/item?id=39414532

it's probably correct to think of functionally all ML models as being stateless. even something like twitter/fb feed - the models themselves remain the same (usually updated 1-2x per month IIRC) - only the data and the systems change.

an illustrative example: say you open twitter, load some posts, then refresh. the model's view of you is basically the same, even the data is basically the same. you get different posts, however, because there is a system (read: bloom filter) on top of the model that chooses which posts go into ranking and that system removes posts that you've already seen. similarly, if you view some posts or like them, that updates a signal (e.g. time on user X profile) but not the actual model.

what's weird about LLM's is that they're modeling the entire universe of written language, which does not actually change that frequently! now, it is completely reasonable to instead consider the problem to be modeling 'a given users' preference for written language' - which is personalized and can change. this is a different feedback to really gather and model towards. recall the ranking signals - most people don't 'like' posts even if they do like them, hence reliance on implicit signals like 'time spent.'

one approach I've considered is using user feedback to steer different activation vectors towards user-preferred responses. that is much closer to the traditional ML paradigm - user feedback updates a signal, which is used at inference time to alter the output of a frozen model. this certainly feels doable (and honestly kinda fun) but challenging without tons of users and scale :)

True, the model is already trained. But it can still be tuned. When I hit the dislike or like button in Claude and leave a comment what I found bad or good I assume my actions are not for nothing, just like a busy box to make me think I'm doing something. If that's true I'll stop doing it. I hope by doing it I'll help tune it and help the next model be better.

I’d point out that this is the “missing link” to a next-gen LLM.

When the scientists figure out how to enable a remembered collaboration that continually progresses results, then we’ll have something truly remarkable.

Regardless of whether the data is actually being used to train an LLM, as long as it's going over the network to an LLM provider, there's the possibility that it will be used. Or exfiltrated from the provider's servers and sold on the dark web. Or a million other things that can go wrong when you provide a 3rd party access to your personal data.

If LLMs are actually here to stay, then we need to keep shrinking the inferencing costs to the point that everyone can run it on the client, and avoid all of these problems at once...

I chat a lot with ChatGPT-4o, usually endless everyday questions like how to replace the screened washer in a kitchen faucet supply line.

It stunned me recently by remembering I had an argon tank, something I had only mentioned in a thread days earlier.

I asked it to recall which conversations these were, for sharing on HN. That conversation was comically stupid (REALLY bad, like "Perplexity" bad) and went nowhere.

OpenAI could of course send all your chat history to a DB and then use that on a future training round. Some of it is free RLHF data that needs a bit of cleaning up. Because the user side of the conversation has clues about whether the answer was correct.

Also are we that far from fine tuning in real time? Maybe next year?

I think while the current model might be what he says the model in 2025 could be vastly different.

The LLMs don't learn during the chat, but what you type into them is being collected, the same as, for instance, your Google searches.

The article is well written. It is precisely because there is no memory that frameworks like langchain exist. If a way to implement LLM with built-in memory capabilities can be found, I feel it will change the game rules of the entire large model market.

Yeah, the difference between "train on this data" and "these interactions might be collected and aggregated across sessions to be included in a future training dataset" is a difficult one to message to most people in software, let alone consumers without that background. By the time you're done explaining how most data is largely garbage, labeling and curating data is difficult and time-consuming, and updates may make things worse such that a rollback is forced, etc. you're up to several paragraphs of explanation and most people won't read that.

Seems like OpenAI needs more data

>All ChatGPT Free users can now use browse, vision, data analysis, file uploads, and GPTs

https://news.ycombinator.com/item?id=40524031

Wow, I figured this would have been the most basic level of common knowledge for LLM end-users by this point. I guess there is still a surprising amount of people out there who haven't jumped on the bandwagon.

Anyone know if there is a LM Studio style tool for training models?

Apart from the forced memories, which are super annoying in ChatGPT.

I dunno. Nowadays you can store the the conversation's history and use different RAG methods to pull relevant pieces into the LLM's context.

Sometimes ChatGPT will ask me which response I like better. That is almost certainly used for something like DPO.

A 'remember everything I say' model would actually be fantastic if ran locally.

I’ve never seen anyone believe the straw man that the author seems to think people believe. People are rightly concerned about their data being trained on for the very real concerns which the author later acknowledges are legitimate risks.

I took Simon's post and pasted it into two different ChatGPT 4o windows. In one, I put "what's wrong with this post, especially with the way it's written?". In the other, I put "what's really great about this post and what does it do well?".

Both sessions worked toward those implied goals. The one where I asked what was wrong indicated long-winded sentences, redundancy, lack of clear structure, and focus issues. Sorry, Simon!

The one where I asked what was good about the post indicated clarity around misconceptions, good explanations of training, consistent theme, clear sections, logical progression, and focused arguments. I tend to agree with this one more than the other, but clearly the two are conflicted.

So, what's my point?

My point is that AI/LLMs interactions are driven by the human's intent during the interaction. It is literally our own history and training that brings the stateless function responses to where they arrive. Yes, facts and whatnot matter in the training data, and the history, if there is one, but the output is very much dependent on the user.

If we think about this more, it's clear that training data does matter, but likely doesn't matter as much as we think it does. It's probably just as equally important to consider the historic data, and the data coming in from RAG/search processes, as making as big of an impact on the output.

Wow. Water is wet. The sky is blue. FFNN models don't update their weights during inference.

Who doesn't already know this?

How did they take this long to figure it out? I would have expected this article when ChatGPT first came out.