One problem is I have no idea what physical address Apple has for me, but I'm sure I have moved at least three times (as many as five) since I gave them that address.
A better solution is require a notarized physical mail in the event of password changes for high-security accounts. Everything else just goes to your email account.