I might be miunderstanding something here, but what complexity here is resolved by making this a framework? Isnt this just:
1. Train model like normal
2. Evaluate model using self
3. Use eval results for DPO finetune
1. Train model like normal
2. Evaluate model using self
3. Use eval results for DPO finetune
The aim is really to give a good base for follow up research / modifications, which I think there will be many for this paper