Hacker News Reader

It’s based on users blind rating the output of two LLMs given the same prompt.

Where GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it.

Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it.

And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.

Typically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.

Reply on news.ycombinator.com