Your argument has a very concrete example. We are a generalist intelligence. And when we encountered chess, we eventually developed specialist intelligence to be the best at chess.
Interesting philosophic topic how to ensure objectives aren't affected too much. You can make this mind experiment even as a human. There are parts of our goal system we might want to adjust, and parts we'd be horrified to, being very reluctant to accidentally risk affecting it.