The population grew from 179M to 281M so the workforce didn’t really double. More like a 50% increase
You can see the tail end (looks like they didn't survey it until 1980) of it here: https://www.bls.gov/opub/mlr/2002/05/art2full.pdf Table 4, look at Black participation rates from 1990 to 2015.
For intuition think of extended families living together, being incarcerated, and being the last hired and the first fired (so out of work longer).
Exactly the answer I was looking for!