Yes

edit:

just to clarify, it would be an observation to say "Most Mathematicians produce their best work when they're young" or "Most gymnasts who make it to the olympics are young"

But to say that X is a young mans game is inherently discriminatory.