> So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

MoE (mixture of experts)?