Direct Nash Optimization: Teaching language models to self-improve

> 7B parameter Orca-2.5 model aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaEval 2.0 (even after controlling for response length), an absolute gain of 26% (7%→33%) over the initializing model. It outperforms models with far more parameters, including Mistral Large, Self-Rewarding LM (70B parameters), and older versions of GPT-4

edit: updated quote w/ more context

While the math seems intimidating, it does not look all too different from SPIN and previous researches. Pretty surprising how effective this is though. The costs here seems to be way higher too (with all the GPT4 calls).

> We also do a brief cost analysis associated with the scaled-up experiment on 600k training inputs. The major line items are the cost of sampling outputs, annotating them with GPT-4 to construct training pairs, and then training the next iteration against those pairs. For _each_ of the six iterations:

> 1. Sampling: it took about 18-24 hours to inference 5 outputs for all 100k examples on 10 8xA100 80GB pods, depending on the average length, costing about $6,000 based on spot pricing.

> 2. Annotation: the average number of prompt tokens sent to GPT-4 for annotation across iterations was about 450M, with an average of about 60M completion tokens, amounting to about $34,000 based on the version of the endpoint we were using.

> 3. Training: ironically, training was the cheapest step, taking only 12-24 hours on two 8xA100 80GB nodes

Do papers like these ever make the data they generate publically avalible or are we expected to pay the same api fees if we ever want to verify the work?

What is the UI for human preference then? Is it still just asking people to pick the best of two options?