Hacker News Reader

They do explain their methodology in some detail in the accompanying GitHub repo: https://github.com/lchen001/LLMDrift

They seem to have been taking to the API directly and requesting the two different model snapshots.

I'm not convinced by their methodology generally. It looks like everything may have been run with temperature 0.1, which I don't think reflects most real-world usage for example.

View on HN

In looking at the paper's continuous mention of "ChatGPT" and the repo README's statement that "You don't need API keys to get started" .. are we sure they weren't using type of tools to talk to the ChatGPT API (via a session token, etc) vs the OpenAI API? I do agree they talk about the API in the paper a lot but I don't see an exact methods statement that they directly accessed the non-ChatGPT API anywhere, unless I'm missing it,

I think the lack of API key note is because that notebook is the one that renders the charts for the paper.

Reply on news.ycombinator.com