Completely agreed, and this has been argued since antiquity. In Plato's Republic:
Socrates: Therefore, calculation, geometry, and all the preliminary education required for dialectic must be offered to the future rulers in childhood, and not in the shape of compulsory learning either.
Glaucon: Why's that?
Socrates: Because no free person should learn anything like a slave. Forced bodily labor does no harm to the body, but nothing taught by force stays in the soul.
Glaucon: That's true.
Socrates: Then don't use force to train the children in these subjects; use play instead. That way you'll also see better what each of them is naturally fitted for.