Because most of those people are not repeat violent criminals? We only need to incarcerate repeat violent offenders to reduce crime noticeably. This is not controversial. The data is linked-to above.
If we incarcerated everyone, it would be even lower.
Reductio ad absurdum
> Because most of those people are not repeat violent criminals