I recently saw Sam Altman bragging about OpenAI's performance on Codeforces (leetcode-like website), which I consider just about the worst benchmark possible.

1. All problems are small- the prompt and solution (<100 LOC, often <60LOC)

2. Solving those problems is more about recollecting patterns and less about good new insights. Now, top level human competitors do need original thinking, but that's only because our memory is too small to store all previously seen patterns.

3. Unusually good dataset- you have tens of thousands of problems, each with thousands of submissions, along with clear signals to train on (right/wrong, time taken etc), a very rich discussion sections etc.

I think becoming 100th best Codeforces programmer is still an incredible achievement for a LLM. But for Sam Altman to specifically note the performance on this- I consider that a sign of weakness, not strength.