To put it in perspective: there are more people who want a job but aren't looking for a job than there are people in the entire US prison system (2.6 million vs 2.4 million).
It would not surprise me if the two are related. Google says there are approximately 20 million felons in the US. It's not difficult to imagine many of them are unable secure employment.