GPT-4: 8 x 220B experts trained with different data/task distributions

As a heavy user of GPT-4 (I'm working on a plugin), reading this felt like a puzzle piece being dropped into place.

Maybe this is just confirmation bias, but yeah, trying to push the model's capabilities is like working with a committee of brilliant minds chaired by an idiot.

Also, I can see why they kept this secret. Competitors just shaved months off their R&D timelines.

The only paper that I could find using an approach with fully separated experts like this is https://arxiv.org/pdf/2208.03306.pdf

the source podcast that this came from: https://news.ycombinator.com/item?id=36407269

Is this an actually confirmed detail or just something George Hotz speculated? How credible is it?

I feel like we should try asking it directly. Maybe it would tell us.