I think part of this is that, obviously it is totally fine to discriminate based on age. Children aren't allowed to work, vote, buy certain items, etc. You're also not allowed certain public offices if you're too young.
Old people are often just as intelligent as their younger colleagues, and often wiser as well. Of course there are exceptions, but there are poor performers at every age bracket for a variety of reasons.
Perhaps people have morphed their internal definition of the word "discrimination" into something that differs from the dictionary and therein lies the objection?