https://www.science.org/doi/10.1126/science.abq1158
For those wondering, yes, these are the same results reported in the February 2022 pre-print here:
https://arxiv.org/abs/2203.07814
As in the preprint the authors report their system's average ranking as top 54.3% in past competitions but the meat and potatoes of their results are in the reporting of test accuracy, which they don't advertise in the abstract- because it's not that good. From the body of the Nature article:
>> With up to 100,000 samples per problem (10@100K), the AlphaCode 41B model solved 29.6% of problems in the CodeContests test set.
So the best test set performance they got was 29.6% with the 10@100k metric. See also Figure 3 in the Nature paper (graphically presenting results listed in Appendix Table A2 in the preprint).
"10@100k" means that their LLM generated millions of programs for each exercise, of which 100,000 (100k) were selected by filtering and clustering and various other heuristics, and of those 100k, 10 were selected to submit as the system's solution.
So 10@100k means to take a few million guesses, then take another 100k guesses, and finally andother 10. And still only get it right 30% ish percent of the time.
This may be enough to rank in the 54% of CodeForces participants, for a system fine-tuned on CodeForces-like data (the CodeContests dataset developed by DeepMind specifically for this task). But it's not enough to claim that AlphaCode "conquers coding, performing as well as humans", per the title of TFA.