Latest GPT-4 training data updated to December 2023 | Hacker News Reader

Latest GPT-4 training data updated to December 2023

platform.openai.com

89 pointsJimmc41445 comments

How are they filtering out posts from existing generative ai models in the training set? Or do they just not bother and train on increasingly polluted datasets?

The same way our understanding of history is polluted by an over-reliance on secondary sources

Thoth’s curse

At least historians understand the implications of a biased source and consider those positions in that context.

This is a great question. I’m curious if anybody has read anything from OpenAI explaining how they’re intending to deal with this.

We will learn whether the degeneration of generative AI leading to model collapse as predicted by some is inevitable (“content is generated as an ‘average’ from mathematical inputs, degenerating because you cannot jump outside the inputs).

https://vue.ai/blog/ai-transformation/the-degeneration-of-ge...

I was so confused when I saw the domain, I asked myself what the Vue framework got to do with this. But visiting the link explained it.

We have truly hit the limit of names in this industry.

So the version number is continuing as MMDD even though it’s a new year? That’s…confusing.

They should have moved to add YY in there.

Makes you wonder about the rest of their systems architecture that we can't see.

It's weird, I use the standard "gpt-4" through the API and it's still stuck on September 2021. Annoying. According to the docs it's supposed to point to the latest version but it's never updated.

Ps I also tried gpt-4-turbo but the API rejects it for some reason. Same with the "numbered" ones listed in the documentation, I only seem to have access to the standard gpt-4 somehow.

Do you use a pinned version in your code?

No just gpt-4

Try the newer pinned versions (sometimes in preview).

'gpt-4-turbo-preview' is the one that always points to the latest version of gpt-4-turbo

Thanks I'll try that!

Which version is the chat? It used to show it, but I no longer see it

You can still ask it:

> when were you last updated?

My last update was in April 2023.

How would we know its telling the truth?

it is explicitly given that in the system prompt https://github.com/spdustin/ChatGPT-AutoExpert/blob/835baae7...

So, its Apr 2023 and not Dec 2023?

Usually they roll updates over several days, so most likely it's currently different for different people.

Can this thing be updated real time?

Probably not right now, the standard process would involve re-running a 'finetuning' part after any update to the underlying model, and while that's far less expensive than the main training, it's probably not something you'd want to do every day.

I'm asking you this since you sound like you might know. At what point in the process do they add in the guardrails/"baby-proofing"? And how do they do it?

There's usually a two or three step training procedure, first training to predict the next word on a huge corpus of text (billions or trillions of words), then possibly some instruction tuning (giving the model question & answer pairs and training on the answer) and then finally RLHF (or RLAIF, DPO etc) where the model is trained to match human preferences. It's this last step that is used to increase the helpfulness & harmlessness of the model, training it to not respond to certain topics.

In general, the core language model is simply trained on a very large amount of unannotated text (which is the most time-consuming and expensive part), but a language model is not directly very useful in the role of e.g. a chat agent, it quite literally tries to continue text and that sometimes is what you want and sometimes isn't.

The second step is fine-tuning the model on a much smaller set of annotated data which specify that it should actually "do something" in its responses and what it should do; it "teaches" it that it should actually answer the questions instead of e.g. continuing on with a list of more questions in the same vein, and most such training sets also "teach" it that for certain questions the appropriate response is a refusal.

If you have the original core model (before that instruction tuning) then you can repeat the same process but instead replace the instruction training set with a different one, so you can "instruct" the model to behave differently. Here is a nice and informative article from Eric Hartford about how he did that to make certain 'uncensored' models - https://erichartford.com/uncensored-models

I’m curious, how come Gemini doesn’t have these knowledge cut-offs?

Why do you assume that Gemini doesn't have these knowledge cut-offs?

If I ask Gemini, for example "Give me three important events that happened during May 13, year 2023.", then it says that this is "in the future" and responds that it can provide some guesses "Based on publicly available information from early February 2023", so that probably is the cut-off date for the model.

However, I would assume that (just like Bing) for certain questions it can pull in extra information from web searches - the model can be old, but if the system puts some retrieved document(s) in the prompt context so that the model can use them for generating the response, it can use that (limited) fresh information as well.

I might have been a stupid assumtion indeed but I assumed it based on the fact that Gemini does not respond with the same shit chatGPT says regarding its knowledge cut-off.

Not entirely related to just GPT-4, but I feel like not many updates will matter for ChatGPT until they completely overhaul the system prompt. This seems to be the reason why ChatGPT has gotten "dumber". https://twitter.com/AlphaSignalAI/status/1757466498287722783

gpt-4-0125-preview has been out for about a month now

The knowledge cutoff seems to be new, at least in the docs.

See this 25 day old screenshot of the docs which says knowledge cutoff April 2023 for the 0125 model

https://www.reddit.com/r/ChatGPT/comments/19fhb6h/new_gpt_4_...

This help article which was updated "this week" also still mentions April 2023

https://help.openai.com/en/articles/8555510-gpt-4-turbo-in-t...

And in the announcement for 0125 they didn't say anything about the knowledge cutoff change

https://openai.com/blog/new-embedding-models-and-api-updates

Did they actually re-release 0125 as a retrained model with newer data, or was that an oversight? The date of Dec 2023 seems to suggest that was always the training time of the snapshot, rather than OpenAI silently re-revving the 0125 model with this doc update being their announcement. But it's a preview model, so maybe anything goes.

This is puzzling. They have either:

1) Replaced a pinned version with a new model (problematic for response consistency), or,

2) Decoupled knowledge cutoff from the model (how!?)

In my experience, these updates don't matter that much. For example, GPT-4 still uses Pydantic v1 syntax (which for the most part is deprecated) even though Pydantic v2 came in June 2023. the information regarding v1 is so much more in GPT-4's training data that "on average" it picks that instead of the new v2 syntax data.

This has been a major pain point for me, because GPT-4 constantly uses @validator, "const", "always", etc. features that don't work on v2.

This is why with future models and their effectively unlimited context size you'll simply concatenate the entire API codebase you're using into the context of your prompt.

I've done similar already, but ChatGPT still tends to rewrite the code to the old API, even if it's correctly using the new API. Ironically, I just had this happen to me while using ChatGPT 4 to use the OpenAI API.

As far as I can tell, pricing per token is not going away anytime soon.

At 100k and above price will naturally come down to be equivalent

Although the attention is really good at focusing on specific information, I've had problems with this approach. It seems like if I ask a task and provide a large enough API it gets distracted or ignores the chunk of text and does its own thing anyway.

Also it's still more expensive to increase the context length

GPT-4 knowledge cutoff is september 2021. GPT-4-Turbo knowledge cutoff was April 2023.

Both models were not able to have any pydantic v2 training data.

have you tried adding a clause that all output must come in pydantic v2 in your prompt? additionally you can add a few common fewshot examples to get it in the pydantic v2 state

I tried exporting the Pydantic documentation page on syntax changes as PDF, feeding that to GPT-4, and then telling it to be careful about Pydantic syntax changes. That helped it use the new syntax, but we can't feed in the entire Pydantic doc so it still didn't get the details right.

In my experience, updates decrease quality because of ever increasing censorship. Improvements are negated really quickly.

Reply on news.ycombinator.com