That is actually the same idea as the (now) popular "Mixture of Experts" approach.