Hacker News Reader

Comment by tellarin | Hacker News Reader

Yes, the models are updated officially around every three months, with a notice you can still use the previous version for a time until it is decommissioned.

Some people claim there are also unannounced changes, but I can't vouch for that.

The daily variation is likely due to temperature. To make the response less repetitive.

View on HN

Wasn't there a study recently that tracked the performance of GPT over time and found significant drop in quality? Did those drops occur at official model changes, or at other times? (i.e. unannounced changes for safety or cost reduction)

I mean, if I was OpenAI, I probably wouldn't make an announcement like "we've just quantized the model and increased our profit margins significantly! The only change on your end will be a slightly dumber model. (Don't worry! Most users won't even notice!)"

This one [1]? That tracks two distinct versions (0613 vs. 0314).

Also, IMO, the tasks they evaluate aren't useful (I rarely want my LLM to tell me whether 17077 is a prime number), and there's room for cherrypicking/survivorship bias. My guess is that OpenAI did something between 0314 and 0613 that shifted focus away from maths to other subjects.

[1] https://arxiv.org/pdf/2307.09009.pdf

I haven't tracked the quality but I do track the performance:

https://gpt-monitor.adamkdean.co.uk/

It fluctuates a lot but you can see trends.

The site linked in the OP is interesting because it takes a picture from GPT every day, so we can see for ourselves if there is any difference with time. We can come back tomorrow and see what it has produced. If it has produces random squigly lines again, we might assume that today's success was just a fluke.

Reply on news.ycombinator.com