The workforce grew, look at the link and table 5. Basically woman and minorities joined the workforce. Worker to population ratio went up and the total number of workers went up faster than the growth in population.
You can see the tail end (looks like they didn't survey it until 1980) of it here: https://www.bls.gov/opub/mlr/2002/05/art2full.pdf Table 4, look at Black participation rates from 1990 to 2015.
For intuition think of extended families living together, being incarcerated, and being the last hired and the first fired (so out of work longer).
Exactly the answer I was looking for!