The reason the focus is on the SSN is because it enables credit. Privacy is important, but so is protecting your finances.
Recently I called to report a lost credit card, for instance, and the operator read through a list of 10 addresses. I had to confirm which ones I'd lived at at some point in my life, in order to verify my identity.