Training examples are unlikely to contain non-faces with high-noseness to the sides and high-eyeness in the center, but plenty of pictures with nose to the left, eye in the center, nose to the right, eye in the center.
So there are many examples which increase the weights for faceness detector for pictures with noses on the sides, eye in the center, but there are only a small number of examples which decrease the weights for the said pictures.