This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?