Hacker News Reader

its moe, each expert tower can be branched from some smaller model.

That's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.