I expected them to have trained a novel network to solve the problems. In fact, it seems they have used OpenAI Codex (a similar network to the one used by Github copilot) to generate the answer.

Their insight was to edit the question-answer pairs to include domain knowledge and solve the questions as python code.

Quite impressive results nevertheless! It makes wonder whether this kind of problem can be solved using a custom-made solution, or is training large networks that can generalize from vast quantity of information the only way. Because if it is, then all these solutions would be monopolized by the large players..