When most of them have at least one hole I recycle them and start over.
For reference, these are good socks in my book: http://www.happysocks.com/uk/combed-cotton-polka-stripe-14.h...
The advantage of having such socks is that, since I normally only wear black, it make me seem slightly less boring. The disadvantage is that it's hard to find really worthwhile socks. The secrets of the trade are:
1. Always be alert. If you pass by a sock shop, go in. I once did it on a first date.
2. If you see socks you like, buy them. Don't say "I'll come back later" or "I'll find them online". You'll soon realise it's a buy-now-or-get-heartbroken affair.
3.the Internet is your friend. Compile a list of brands who are more likely to produce good socks and online shops who sell them.
3.5 Pro tip: Swedish brands are very colourful-sock friendly.
4. Visit those online shops regularly to see if they have anything worth buying.
5. You can never have enough socks, so if you see qualifying socks, buy them. You can always store them and use them at a later date.
6. Never buy more than one pair of the same design. The only exception is if there is a pair you see you exceptionally like, in which case you can buy two pairs, but store one pair to when the first pair is too worn out to wear.
Edit: formatting.
Edit 2: this obviously also solves the problem of having to throw good socks out as their general population get worn out.
I don't have a formal rotation system, though. Just drawing at random from the drawer seems to produce approximately equal wear.
Of course I also own few other colors for special occasions and sport socks, they go to separate drawer.
I'd agree khaki pants with brown shoes would be weird with black socks
If you don't care about fashion, it doesn't matter, though