As you mention, it's easily determinable, and it has the fortune of being an existing ID number system already connected to everyone.
Allowing everyone the ability to tell people apart based on a number is infinitely better than matching names.
In my country we use our ID numbers for everything, dentist appointments, vehicle inspections.. if you call any business at all it will likely be one of the first things you're asked for.
The number should be an identifier for a person, not a security measure. If you just accepted the fact that SSNs are insecure and that by knowing someone's SSN does not necessarily mean you are that person, then the US could use them more effectively, cut down on a significant number of errors in every industry, and actually reduce identity theft.
I can see a point in having an identification system that spans the country and is easy to use, but SSNs are not (and should not be) it. It's also something unlikely to properly happen in the US due to the state/federal powers separation. (Which is why we have the kludge of so many businesses using SSN for the purpose in the first place...)
Prior to the IRS requiring a SSN for claiming dependents, many people did not get a SSN until they started working; parents did not apply for them at birth. Most of the people I went to high school with, got them in high school.