Traditional fashion wisdom is that socks should match pants. If you match your pants to your shoes, it makes it look like you're wearing high-tops. It draws attention away from your shoes when you want to be drawing attention toward your shoes.

If you don't care about fashion, it doesn't matter, though