To add to this, prior to the internet it really wasn't that bad of a "password". Once upon a time vacuuming up batches of SSNs for nefarious purposes wasn't a realistic attack scenario, let alone a "just assume every criminal has your SSN" one.
It's mostly a terrible password because it's immutable if it weren't it's a quasi random 9 digit code that's hard to map from a person's current information back to what their SSN is.
If you know when and where someone was born you can figure out the first 5 digits.