Would it be correct to summarize that the general conceptual shift is optimizing MOEs on more specific smaller tasks? It smells like borderline overfitting to me for some reason.
that MoEs are better for a given compute budget has been known for a while.
They also mention in the source tweet that the quality of the base model is important, which is not really accounted for in the $30 figure.
'rl on sparse rewards' = RL objective - reward is given by deterministic evaluation, reward is passed back as sparse signal. there is no 'ground truth' data to compare against.