It took me a long time to figure this out as a child, since nobody I asked knew the answer! Is it something about the vertical of the mirror being different than the horizontal? No, if you rotate the mirror 90 degrees it still inverts left/right and not top/bottom. Is it something about the Earth that makes horizontal different from vertical? No, if you look into a mirror while your body is in horizontal position, it still inverts left/right, and not top/bottom (viewed from your body). Is it something about our two eyes being on a horizontal line instead of a vertical line? No, if you close one eye it still works all the same. If you trace out how the light goes, you'll see that there is nothing different about horizontal vs vertical. Yet clearly a mirror inverts things horizontally but not vertically. How can this be?

The real answer is that we perceive this because we are used to turning in the horizontal plane. If you see yourself in the mirror and you raise your right hand, the guy in the mirror is raising his left hand. But how do you know that it is his left hand? Mentally you place yourself in his position by rotating yourself and then you check which hand it is, but by doing this rotation your left hand ends up in the position of your right hand in the mirror. So even though the alignment of you and your mirror image appears to be perfect on first sight, it actually isn't. If we were creatures that are used to only turning in the vertical plane, then we'd perceive the same image as top/bottom inversion. You'd mentally rotate yourself vertically, and then instead of your left hand ending up where you see your right hand and vice versa, your head would end up where you see your feet and vice versa.

The fact that our bodies are roughly symmetrical in one direction but not in the other makes this a bit non obvious unfortunately. It may be easier to understand this with an object that's not symmetrical in any direction. If you have a piece of paper with some word written on it, and you look at it via a mirror, why is the word flipped from left to right and not from top to bottom? This question has the same answer: because we're used to turning things in the horizontal plane. To show the paper to yourself via the mirror, you rotated the paper so that the text is now facing away from you, towards the mirror. But how did you rotate it? You rotated it by turning it in the horizontal plane. If you had rotated it in the vertical plane, the word would be flipped vertically. It's just that if you do this nobody is surprised that the word ends up flipped vertically, because that's not how our psychology works.

Print out a word in a very large font on a piece of paper. Now, hold the blank side up to a light and try to read the word - it will be backwards. This is because you're reading the paper back to front.

Our mind makes it a left to right inversion because we first try to think of it as a rotation. Look in the mirror while holding a hat in your left hand and a shoe in your right (the objects are arbitrary - substitute where necessary). Now, look in the mirror. The hat will be on YOUR left and the shoe will be on YOUR right. There's no left to right swap. The reason that I capitalized "YOUR" is that I don't want you considering whether it's on the left or right of your REFLECTION. That's where the left to right inversion mistake comes from - we think the shoe is on the reflection's left. Normally, when we're facing another person, that person has rotated 180 degrees to face us. That puts her left on our right and vice versus. We mentally assume that's the case with the reflection in the mirror - that the reflection's right hand is on our left. However, the reflection wasn't rotated 180 degrees to face us. It was flipped front to back. The reflection's left hand is on our left, just as you would expect from something that has not been flipped left to right.

Since left and right and up and down are both the same, it's essentially as though you're looking through the back of the object. Imagine if you made a sign clear and looked through the back. Even though no reversal has taken place, the text looks backward. This is why things appear reversed.

If you are confused about how left and right are not actually switched, I would say the easiest way is to think in absolute terms, not relative. That is, don't think about left and right, think about east and west. Face north and paint your east (right) hand red. Now when you look at a mirror (you're still facing north), your east hand is red, but the reflection's east hand (think in real world terms here) is also red. That's because the reflection's horizontal aspects have not been reversed.

Richard Feynman enjoys discussing this: http://www.youtube.com/watch?v=msN87y-iEx0&gl=CA

Ambulances have the word Ambulance flipped backwards, not flipped upside down.

This intuitively seems to be a major break of symmetry, why would a mirror have a horizontal preference?

It isn't the mirror with the preference, but the vehicle. To understand why, consider an ambulance facing you head on with the word "AMBULANCE" printed on it and the paths it could take to show up in your rear-view. It could move along a lateral half-circle or a vertical one (or something in-between). If it went along a vertical half-circle it would end up upside-down and behind you; since it goes along a lateral one it also gets laterally-flipped, but because cars have left to right mirror symmetry , you don't notice (except for the words ambulance).

We have a huge bias evolutionary bias towards that type of lateral symmetry (e.g. in animals, other humans) and to things moving along a lateral plane while maintaining vertical orientation.

I do not know whether this will help or confuse, but it is a different way to look at it:

Look into a mirror with your nose pointing North. Your left hand will point West, and your right hand will point East.

Your mirror image has a nose pointing South, a left hand pointing West (!) and a right hand pointing East (!)

You know it is his left hand pointing West because it moves when you move your left hand. Also, you may wear a ring or have distinctive features on one hand.

You know that left hand is pointing West because you can see the sun set in that direction.

So East and West stayed where they were. Also, up and down still are where they were; rain still falls on your head, not on your chin.

The directions that changed are North and South.

The effect of that is that, from the point of view of the mirror image, East and West seem to have switched sides. The mirror image is looking South, and yet, West is to its left and East to its right.

The real headscratcher comes when you realise that all reversions are equivalent disregarding rotation. That is, reversing left/right is the same as reversing up/down and rotating around z.

You can actually get a up/down mirroring effect (intuitively speaking) by standing on a mirror and holding a piece of paper perpendicular to it. It will look up/down reversed, not left/right.

Feynman explains it best: http://www.youtube.com/watch?v=msN87y-iEx0&t=46