For ID=3, it shows o1 getting it wrong, but it seems to have succeeded? It did add a space between Tinker and bell, but that is the canonical way of spelling the character apparently.
(That just one caught my attention because I was curious what challenge o1-mini got correct that o1 did not.)
Thanks, fixed. (Spaces rebuilding.) We have manually combed labelled-wrong answers and tweaked the predicates that check correctness. Sorry we missed this one.
I thought maybe I was just missing something silly.. was hesitant to comment without additional coffee. And no problem, thanks for sharing this!
Is there some standard markup assumed for LLMs? I just looked at a couple and saw markdown-like text (which I expected), but also "$\boxed{foo}$". Looks like LaTeX, but is there some subset of LaTeX LLMs are assumed to know/use?