Taalas is developing this, but not for Frontier class models. I hope that if we can least get the easy 80% of work done on that sort of hardware, we can greatly reduce the demand for GPUs, HBM and energy to some extent.
There was a paper a while back that showed top-K selection like that with tiny models was able to reliably solve some 1M-step Tower of Hanoi when no frontier model could. Very big level up in capability just from horizontally scaling compute.
If I can make my small fast AI model just a tiny, tiny bit more capable, and still run 100 of them or 1000 and run an evaluation model on top of that, the overall system capability will scale quickly with tiny increases in base model intelligence.
So i guess maybe they currently try to solve a very hard problem with a small focused group before scaling or they are dysfunctional.
Also Llama 3.1 8B is a dense model AFAIK and they are fast by nature. As there are not a lot of dense models these days i could imagine that they try to optimise for MOE models.