Yeah, and people within earshot are not the issue, it’s the place that has thousands of SSNs getting hacked that the issue, so there is no reason to be secret about it.
To add to this, prior to the internet it really wasn't that bad of a "password". Once upon a time vacuuming up batches of SSNs for nefarious purposes wasn't a realistic attack scenario, let alone a "just assume every criminal has your SSN" one.
It's mostly a terrible password because it's immutable if it weren't it's a quasi random 9 digit code that's hard to map from a person's current information back to what their SSN is.
If you know when and where someone was born you can figure out the first 5 digits.