Hacker News Reader

Comment by dangond | Hacker News Reader

> The idea behind GPT Unicorn is quite simple: every day, GPT-4 will be asked to draw a unicorn in SVG format. This daily interaction with the model will allow us to observe changes in the model over time, as reflected in the output. Is it useful to do this every day? Correct me if I'm wrong, but my understanding is that OpenAI does not update the models available in production incrementally on a day-to-day basis.

View on HN

They do update the model in the background, although I'm not sure how often or how much they update it. To avoid issues with this practice they offer gpt-4-0314 which says this in the documentation:

"Snapshot of gpt-4 from March 14th 2023. Unlike gpt-4, this model will not receive updates, and will only be supported for a three month period ending on June 14th 2023."

Unfortunately this experiment is using the frozen snapshot model gpt-4-0314 instead of the unfrozen gpt-4 or gpt-4-32k models, so any differences are literally 100% noise. This would be a somewhat interesting experiment if someone were to use an unfrozen model, though. I do appreciate the author for captioning the images with the exact model they used for generation so that this bug could be caught quickly.

[0]https://platform.openai.com/docs/models/gpt-4

Author here: these images are using `gpt-4` but I'm recording the specific model that OpenAI use with each result. As the incremental updates come out, that will change (without requiring me to change anything.)

Similarly the quality of the model can't be judged with a single sample. These end up canceling out.

I updated my blog post with clarifications but I expect to see two types of changes over time, the variance of output, and the quality of output.

Reply on news.ycombinator.com