Does anyone know what Figure 8 at the end shows?
It looks like each expert is used interchangeably with no clear pattern. And earlier they say "Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic."
So then, what is the point of the "expert"?
Could this extra performance just be through the 8 expert architectural design, and not be based on the underlying training material? For example, if all 8 experts were ArXiv papers, would the performance be different?