1. How hard it is to grok code that I've never seen before
2. How hard it is to grok my own code from 6 months ago
Sometimes it may take me a day or more to fully get my head around a new piece of code I am working with. I could not give a thorough AND fair review of another person's code without internalizing it to a satisfactory degree.
There is also the issue of software as art vs engineering. Quite often I will take issue with the style or approach of another person's coding simply because I would have done it differently had I done it myself. But that does not mean that their code is wrong.
All things considered, I think code reviews can be helpful in certain situations, but I think there are many pitfalls which must be avoided in order for them to justify the cost.