inb4 this technique is subsumed into the next MoE model release
LLMs are evolving so fast I wouldn’t be surprised if this technique was not needed in <6 months
LLMs are evolving so fast I wouldn’t be surprised if this technique was not needed in <6 months