What you're describing is one possible resolution and part of what's interesting. A very big open question is "which, if any, of the known factors are wildly rarer than we think?". Another possible resolution is that most of our estimates are actually ok and there's some other variable that's going to whack us on the head at some point.