The typical voter didn't take a class in statistics. I know I didn't.
As far as I know, 99.999999..% of the voters are not at the median, so it escapes me why anyone should be concerned with the person in the middle.
Also, what if the number of voters is even? Then there is no median voter, it's just an abstraction.
What is a median in multiple dimensions, anyway? Couldn't it easily be far away from any individual? There was a thing that's probably been on HN multiple times about how virtually nobody is "normal" in, say, five or more characteristics at once.