Hacker News Reader

Comment by behnamoh | Hacker News Reader

In my experience, these updates don't matter that much. For example, GPT-4 still uses Pydantic v1 syntax (which for the most part is deprecated) even though Pydantic v2 came in June 2023. the information regarding v1 is so much more in GPT-4's training data that "on average" it picks that instead of the new v2 syntax data.

This has been a major pain point for me, because GPT-4 constantly uses @validator, "const", "always", etc. features that don't work on v2.

View on HN

GPT-4 knowledge cutoff is september 2021. GPT-4-Turbo knowledge cutoff was April 2023.

Both models were not able to have any pydantic v2 training data.

This is why with future models and their effectively unlimited context size you'll simply concatenate the entire API codebase you're using into the context of your prompt.

I've done similar already, but ChatGPT still tends to rewrite the code to the old API, even if it's correctly using the new API. Ironically, I just had this happen to me while using ChatGPT 4 to use the OpenAI API.

As far as I can tell, pricing per token is not going away anytime soon.

At 100k and above price will naturally come down to be equivalent

Although the attention is really good at focusing on specific information, I've had problems with this approach. It seems like if I ask a task and provide a large enough API it gets distracted or ignores the chunk of text and does its own thing anyway.

Also it's still more expensive to increase the context length

In my experience, updates decrease quality because of ever increasing censorship. Improvements are negated really quickly.

have you tried adding a clause that all output must come in pydantic v2 in your prompt? additionally you can add a few common fewshot examples to get it in the pydantic v2 state

I tried exporting the Pydantic documentation page on syntax changes as PDF, feeding that to GPT-4, and then telling it to be careful about Pydantic syntax changes. That helped it use the new syntax, but we can't feed in the entire Pydantic doc so it still didn't get the details right.

Reply on news.ycombinator.com