In light of last week's fiasco with Reflection (https://venturebeat.com/ai/new-open-source-ai-leader-reflect...), I hope the community has a newfound enthusiasm for independent testing!
This is extremely exciting news if true, so I'm eager to have it either confirmed or questioned. The one thing I hope we won't be doing is accepting SOTA evals from open-sourced models at face value.