I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.

I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.

It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly.

I think what you’re describing is the closest approximation we reasonably have right now though.