> Given enough SSNs as training data, could someone make an ML model that churns our mostly valid SSNs?

Most SSNs are deterministically derived. Given a single valid SSN, you can often generate millions of valid preceding values.

Don't you need at least one other piece of identifying info to make that useful? Like, I could churn out mostly valid gmail passwords all day, but I have no idea which users have those passwords.

No need to get that involved:

for (let i = 0; i < 1000000000; i++) console.log(i);

Every single valid social security number, and several (e.g. 000-00-0000) that probably aren't. You can store the whole thing in about a gig.

Worse. You can make the ML model spit out the SSNs it was trained on. That's a problem when you can't manually curate billions of documents. If you didn't look, you wouldn't even know they were there.