Qwen MoE[0] showed promise for a small size. I hope they spend more time in them.

[0] https://qwenlm.github.io/blog/qwen-moe/