Think of it this way; there is less information in a blurred image (less colour, less lines, less areas). You can not* conjure information out of thin air thus making the unblurred image.

* Recent advances in AI actually make this possible to an extend. The AI delves into it's massive memory and extrapolates a likely image/face.