Iterative reasoning preference optimization

Create chain-of-thought candidates with LLM, form preference pairs depending on whether the answers are correct, train using DPO and NLL, then repeat the cycle.

Apparently this takes Llama-2-70B-chat from 55.6% to 81.6% on the GSM8k benchmark