I think that unless you have a specific reason for restricting them, you simply shouldn't restrict them, since any restrictions on the character set makes all passwords weaker. Personally, I expect all printable ASCII characters to be acceptable as part of a password and get irritated when some of them aren't accepted.