This was interesting to see happen live on X

I think putting this on the llm is a bit generous. Their results were apparently 30x above the theoretical maximum, according to gpu master Tri Dao, so there was also a lack of understanding on what was possible with CUDA. See:

https://x.com/tri_dao/status/1892610951662153945

Also see this thread by Lucas Beyer: https://x.com/giffmana/status/1892510741242036468

One of the greatest skills in research is to remain skeptical of one’s own results, especially when they are exceptional. They chose to pull the trigger, and release too quickly.

This can happen in any setting, not just codegen, e.g. inadvertently training on the test set. Science is a slow ascent: if it looks too good to be true, you probably just have a bug.