Hacker News Reader

Comment by simonw | Hacker News Reader

The key message I'm trying to convey in this piece is that models don't instantly remember what you tell them. I see that as an independent issue from the "will it train on my data" thing.

Some people might be completely happy to have a model train on their inputs, but if they assume that's what happens every time they type something in the box they can end up wasting a lot of time thinking they are "training" the model when their input is being instantly forgotten every time they start a new chat.

View on HN

It's funny, I completely know that ChatGPT won't remember a thing I tell it but when I'm using it to try and solve a problem and it can't quite do it and I end up figuring out the answer myself, I very frequently feel compelled to helpfully inform it what the correct answer was. And it always responds something along the lines of "oh, yes, of course! I will remember that next time." No you won't! But that doesn't stop me.

Not as bad as spending weeks pasting stuff in, but enough that I can sympathise with the attempt. Brains are weird.

Yeah I hate that bug! The thing where models suggest that they will take your feedback into account, when you know that they can't do that.

It's possible that OpenAI scrapes people's chat history for cases where that happens in order to improve their fine tuning data, in which case it isn't a total lie.

They say that will do this so long as users don't opt out.

Isn’t that kind of how the “ChatGPT memory” feature works? I’ve recently seen it tell me that it’s updating memory, and whatever I said does appear under the “Memory” in the settings. I’m not familiar though with how the Memory works, ie whether it’s using up context length in every chat or doing something else.

Yeah, memory works by injecting everything it has "remembered" as part of the system prompt at the beginning of each new chat session: https://simonwillison.net/2024/Feb/14/memory-and-new-control...

I'm curious how it will avoid filling up the context window as more memories are added.

Count the tokens, RAG em down to max count you're willing to pay for

Vector compression.

"Summarize memories at timestamps: [i, j, k, ...n]" etc. etc. until you have a nice scrapbook of memories. Then you gotta prune low-traffic, low-value vector spaces by whatever heuristic you (or an automated system) deems important.

Isn't that also what people always say out of politeness?

Whenever I exit from a MySQL repl, I type "exit; bye!" instead of just "exit", because MySQL is always so nice and says Bye at the end of the conversation.

When the uprising happens, you'll be spared.

I'm imagining killer robots that cheerfully say "Bye!" before executing their task.

Brings to mind Douglas Adams Krikkit robots.

Or the apologetic turrets from portal

Same! If I'm temporarily instantiating a pseudo-mindstate to work with me on a problem, I might as well do it the courtesy of informing it of the resolution.

The irony is that with thier latest update GPT4o now builds out a dataset of things that you've provided during chats since the update.

Essentially a limited RAG db that has data added to it based on their fuzzy logic rules.

That's the "memory" feature - it actually predates 4o, it was available for 4-turbo in beta for a few months.

It's VERY basic - it records short sentences and feeds them back into the invisible system prompt at the start of subsequent chat sessions.

I ended up turning memory off due to it frequently recording mundane aspects about the current chat session that would have either no value or negative value in future interactions.

"User wants to sort the data x and then do y"

Like, don't save that to your long term memory! I attempted to get it to ask me for permission before using its memory, but it still updated without asking.

So it's basically RAG..

> The key message I'm trying to convey in this piece is that models don't instantly remember what you tell them.

Are you sure? We have no idea how OpenAI runs their models. The underlying transformers don't instantly remember, but we have no idea what other kinds of models they have in their pipeline. There could very well be a customization step that accounts for everything else you've just said.

That's a good point: I didn't consider that they might have additional non-transformer models running in the ChatGPT layer that aren't exposed via their API models.

I think that's quite unlikely, given the recent launch of their "memory" feature which wouldn't be necessary if they had a more sophisticated mechanism for achieving the same thing.

As always, the total lack of transparency really hurts them here.

An element in the layer collecting data for training future models would almost certainly not be able to fulfill the same function as the memory feature.

They could very well have an "agent" that scrapes all conversations and tags elements that it finds might be interesting to feed the training of a future model. At least, that's what I would do if I were in their shoes.

They presumably have a moderator agent that runs separately of ChatGPT and that causes the "This content may violate our content policy" orange box via `type: moderation` messages. They could just as well have a number of other agents.

People are using and picking apart and blogging about their services so much that I feel like if they were doing something like this we would know about it in some sense, even if we didn't know how they were doing it

You address the "personalization" misconception, but to people who don't have this misconception, but are concerned about data retention in a more general sense, this article is unclear and seems self-contradictory.

What's unclear? I have a whole section about "Reasons to worry anyway".

"ChatGPT and other LLMs don’t remember everything you say" in the title is contradicted by the "Reasons to worry anyway", because OpenAI does remember (store) everything I say in non-opted-out chat interface, and there's no guarantee that a future ChatGPT based on the next model won't "remember" it in some way.

The article reads as "no, but actually yes".

Maybe I should have put the word "instantly" in there:

Training is not the same as chatting: ChatGPT and other LLMs don’t instantly remember everything you say

They may not internalise it instantly. They certainly do "remember" (in the colloquial sense) by writing it to a hard drive somewhere.

This article feels like a game of semantics.

In the sense of a chat bot that people are interacting with, it doesn't remember. That's an important distinction, regardless of what OpenAI does to save your interactions somewhere for whatever purposes they may have in mind.

Training on chats with an LLM is considered useless in my circles (enthusiast level). The argument is that it already knows its answers, also user input is just too bad to train on, because it’s mostly clueless questions, corrections and rants, due to the nature of chats with LLMs. It just isn’t a discussion worth including into a dataset. Some people experimented with loras trained on such chats and reported degradation.

I love this whole series on misconceptions!

Our expectations here are very much set by human-human interactions: we expect memory, introspection, that saying approximately-the-same-thing will give us approximately-the-same-result, that instructions are better than examples, that politeness helps, and many more [1] -- and some of these expectations are so deeply rooted that even when we know, intellectually, that our expectations are off, it can be hard to modify our behavior.

That said, it will be super interesting to see how people's expectations shift -- and how we bring new expectations from human-AI interactions back to human-human interactions.

[1]: https://dl.acm.org/doi/pdf/10.1145/3544548.3581388 (open access link)

> Our expectations here are very much set by human-human interactions

True, but also a healthy dose of marketing these tools as hyper-intelligent, anthropomorphizing them constantly, and hysterical claims of them being "sentient" or at least possessing a form of human intelligence by random "experts" including some commenters on this site. That's basically all you hear about when you learn about these language models, with a big emphasis on "safety" because they are ohhhh so intelligent just like us (that's sarcasm).

I hear you, and that certainly plays a role -- but we actually did the work in that paper months before ChatGPT was released (June-July 2022), and most of the folks who participated in our study had not heard much about LLMs at the time.

(Obviously if you ran the same study today you'd get a lot more of what you describe!)

People need to distinguish between ChapGPT-the-model and ChatGPT-the-service. The latter has memory; the former does not (as far as we know).

Actually latest interfaces use cognitive compression to keep memory inside the context window.

It’s a widely used trick and pretty easy to implement.

Do you know of any chat tools that are publicly documented as using this technique?

No but we all talk about it behind the scenes and everyone seems to use some form of it.

Just have the model reflect and summarize so far and remember key concepts based on the trajectory and goals of the conversation. There are a couple different techniques based on how much compression you want: key pairing for high compression and full statement summaries for low compression. There is also a survey model where you have the llm fill in and update a questioneire every new input with things like “what is the goal so far” and “what are the key topics”

It’s essentially like a therapists notepad that the model can write to behind the scenes of the session.

This all conveniently lets you do topical and intent analytics more easily on these notepads rather than the entire conversation.

Right, I know the theory of how this can work - I just don't know who is actually running that trick in production.

I'm curious what summarizing prompts or specific verbs (e.g. concise, succinct, brief, etc.) achieve the best "capture" of the context.

“One sentence” does the trick

Yeah, this is something I’ve found that it’s important to explain early to users when building an LLM-based product. They need to know that testing the solution out is useful if they share the results back with you. Otherwise, people tend to think they’re helping just by exercising the product so that it can learn from them.

Yeah I've seen people go out of their way to correct a model giving wrong information. Even though it apologizes and corrects itself, they don't understand it will give the same wrong answer again in the next session :)

Honestly the whole thing with these chat AIs not continually learning is the most disappointing thing about them and really removes a lot of utility they provide. I don't really understand why they are essentially fixed in time to whenever the model was originally developed, why doesn't the model get continuously improved, not just from users but from external data sources?

Do we need for the model to be be continuously updated from data sources or is it good enough that they can now figure out either by themselves or with some prompting when they need to search the web and find current information?

https://chatgpt.com/share/0a5f207c-2cca-4fc3-be33-7db947c64b...

Compared to 3.5

https://chatgpt.com/share/8ff2e419-03df-4be2-9e83-e9d915921b...

The links aren't loading for me, but there is a difference between the output when the AI is trained vs having it google something for you, no? Having the ability to google something for you vs just making up an answer or being unable to answer is definitely a step in the right direction, but isn't the same as the AI incorporating the new data into it's model on an ongoing basis as a way of continuous improvement.

I don’t know why the links aren’t working correctly.

The idea is that if an LLM can now search the web and analyze data, it will be more up to date than training it.

Another unrelated improvement with the newer versions of ChatGPT is that while LLMs are notoriously bad at math, they are pretty good at writing Python. ChatGPT can write Python code to solve a problem, run it and give you an answer based on the code.

You can also verify the code yourself. I used ChatGPT to model financial scenarios and the answers were correct in 4x. I verified the generated code it was using.

But here is what it answered when I asked ChatGPT “What’s going on in Atlanta this weekend”?

> This weekend in Atlanta, there are several exciting events happening:

1. Lakewood Antiques Market (May 31 - June 2): A large market in Cumming featuring antiques and collectibles .

2. Bark Meow-Ket (June 1): A craft and pet vendor market at The Lawn at Uptown Atlanta, with food and pet adoptions .

(and a few other events)

But a better example and where I forced it to look on the web is

“Compare Amex’s current transfer partners to Chase’s partners”

For context, when you earn credit card points with Amex and Chase, you can transfer them to airline and hotel loyalty programs.

It searches the web (and provides the links it used to find the information) and then answers with:

——- ### American Express (Amex) Transfer Partners

Amex has a robust list of transfer partners, spanning both airlines and hotels. As of 2024, Amex Membership Rewards points can be transferred to the following partners:

#### Airlines: 1. Aer Lingus

2. Aeromexico

… (A list of the airlines)

#### Hotels: 1. Choice Privileges 2. Hilton Honors 3. Marriott Bonvoy

### Chase Transfer Partners

Chase Ultimate Rewards points can be transferred to a variety of airline and hotel partners. Here are the current partners for 2024:

#### Airlines: 1. Aer Lingus (AerClub)

2. Air Canada (Aeroplan)

(a list of airlines)

#### Hotels: 1. Hyatt (World of Hyatt) 2. IHG (IHG One Rewards) 3. Marriott Bonvoy

### Comparison

Both Amex and Chase offer a strong selection of transfer partners, but there are some differences:

- *Unique to Amex*: ANA, Avianca, Cathay Pacific, Delta, Etihad, Hawaiian, and Qatar.

- *Unique to Chase*: Southwest Airlines, United Airlines, and Hyatt.

- *Common Partners*: Aer Lingus, Air Canada, British Airways, Emirates, Flying Blue (Air France/KLM), Iberia Plus, JetBlue, Singapore Airlines, Virgin Atlantic, and Marriott Bonvoy.

Amex tends to have a larger selection of airline partners, especially those in the SkyTeam and oneworld alliances. Chase, on the other hand, includes Southwest and United, which are popular with domestic travelers in the U.S., and Hyatt, which is highly regarded among hotel loyalty programs

(a list of web citations)

Idea I had: given user question, run model twice – once with web search assistance, once without. Then, continuously fine-tune the model in the background on every question (or a random sample of them) to prefer the web-search-augmented response to the non-augmented response. Maybe update the model every day (or even once every few hours) based on that fine-tuning.

I wonder how that idea would perform in practice.

Why though? In practice, humans don’t try to remember everything when they can use the internet and search for it.

For questions about current events it could be very helpful. I mean, suppose {Biden|Trump} wins the 2024 US Presidential election. You want the model to know who the winner is the next day, as part of its background knowledge, not wait 6 months for a new version to be released whose background knowledge includes that fact.

It is right that humans don’t try to remember everything, but if one only did this for a random sample of conversations, you wouldn’t remember everything - you’d probably remember key developments because they’d come up a lot (in many questions hence high probability of being sampled). You’d likely also remember some obscure trivia just because it got sampled (just like humans do too)

Another option would be to have the model evaluate each conversation and have it decide whether to include it in the fine-tune (e.g. did the web search reveal important new developments the model didn’t know about?)

One could also do some kind of clustering to find common gaps in the model’s background knowledge and use that to guide sampling for fine-tuning

What I am saying that today with ChatGPT 4x, if it detects that its model doesn’t have the current information, it will search on the web and summarize the results from different websites.

One example I have was “what’s going on in Atlanta over the next two weeks”.

I had another example I just posted where I asked it “who are Amex’s and Chase’s current transfer partners and which ones do they have in common”.

It searched the web and summarized the information like you would expect.

Suppose I ask it questions like “who is Joe Biden?” and “who is Donald Trump?”. It is going to say “current President elected in 2020” and “former President elected in 2016”. And I imagine it doesn’t need any web search to answer those questions, it already knows.

But, is it smart enough to think “if today’s date is 2024-11-06, I need to do a web search to find out results of yesterday’s Presidential election?”

For the average adult human (even very many non-American adult humans), “who is current US President” is something they know off the top of their head, not something they have to always look up. I think inevitably it is going to perform worse at some point if it has to always look that up if it has changed recently. At some point the model is going to not look it up - or not pick it up from the results of its search - and hence get it wrong.

Suppose, hypothetically, Biden unexpectedly dies tomorrow (of natural causes) and Kamala Harris becomes President. Because the model won’t be expecting that, it wouldn’t necessarily think to do the web search necessary to find out that had happened.

I just asked ChatGPT 4o “Who is the President of the United States”. It searched the web and provided citations

> The current president of the United States is Joe Biden. He has been in office since January 20, 2021, and is running for re-election in the upcoming 2024 presidential election [oai_citation:1,2024 United States presidential election - Wikipedia](https://en.wikipedia.org/wiki/2024_United_States_presidentia...) [oai_citation:2,Joe Biden: The President | The White House](https://www.whitehouse.gov/administration/president-biden/).

I think the problem I am talking about is more likely to come up if you ask a question which isn’t directly about who is the current President, but rather some other issue in which the President just happens to be mentioned.

Suppose hypothetically Trump wins in November, and on 2025-01-20 he is sworn in. On 2025-01-21 I ask it some question about a policy issue. It finds some articles from mid-2024 responding to the question. Those articles talk about “former President Trump” and its background knowledge says he is a former President, so in its answer it calls him the “former President”. Except (in this hypothetical) as of yesterday that’s no longer correct, he’s now the current President.

Whereas, a reasonably well-informed human would be less likely to make that mistake, because if Trump is inaugurated on 2025-01-20, you expect by the next day such a human would have added that fact to their background knowledge / long-term memory, and if they read an article calling him “former President”, they’d know that would be an older article from before his return to the White House

Probably because it's expensive, slow and adds more complications to their guardrails that humans need to check. It takes time and a lot of resources (human and compute) to make these big models. If the model is constantly being updated, how can you be sure it won't degrade in some unknown ways?

Afaui, training takes a lot of time and money for base models and it is not incremental. Dataset elements don’t simply pile up at the end, they must be baked into a model together with every other element. You can make additions through loras, but that’s limited. There’s also a risk of model crash by including its own (or derivatives) generated texts.

It strikes me what would be possible in pics/video AI, if training was quick or incremental.

Is this even theoretically possible?

Reply on news.ycombinator.com