To make things simple, yes, that's is. White "scrambles" the light, the correct term is "diffuses" but that's the idea.
Maybe quantum physicists will disagree but the difference between a mirror and a white surface is that a mirror is perfectly flat and therefore will reflect light at always the same angle. It may absorb some of it, or even most of it (black mirror!) but what it reflects, it will do it always the same way.
A white painted surface can be thought as a surface covered in tiny mirrors, all pointed in a different direction, and depending on which tiny mirror a ray of light will hit, it will be reflected in a different direction. With large pieces, like with a broken mirror, you will still see (as you said) scrambled images. But because the pieces are so tiny, you won't be able to make out an image and you will only see white. A glass pane can look like a mirror, but reduce it to a powder and it becomes white. It has nothing to do with absorption, you can have a 1% reflective mirror, and 100% reflective diffuse white.
The way the "tiny mirrors" that make up the white paint are oriented give out its general look. Have them oriented mostly with the surface and you get a glossy finish, have them randomly oriented in many directions and you get a matte finish (this is called specular reflection btw).
I've seen mirrors be called "smart white", it is white because it reflects most light in most wavelengths, and it is smart because it keeps the rays of light oriented.