It's always baffled me that the pitcher doesn't get to decide what pitch to throw.
The big reason why pitchers don't signal to the catcher is because pitcher and catcher must both know what pitch is being thrown beforehand and the batter can see the pitcher but not the catcher, so it's 1000% easier to keep the pitch secret when the catcher calls the pitches.
However, the catcher calling the pitches is beneficial because it allows the pitcher to better concentrate on the act of pitching.