There were very few long term prisoners in medieval society, not because of a progressive outlook but because most crimes were punished with some form of torture, execution, or both. Also, many things were crimes that now are thought of as basic human rights. You can have small prison population if your are willing to truly brutalize your population.
China performs about half of the execution in the world every year, is known to torture dissidents, and I doubt it reports those being re-educated as prisoners. I would hardly consider that more just than America.
A society can also have a small inmate population and be unjust because enforcement is rare, or selective either due to corrupt or ineffective law enforcement. Additionally if acts of vigilantism are common in a society you can have also have less prisoners since "justice" is found in the streets.
I think there are reasonable critiques of the American prisons, criminal justice system, and especially for profit prisons, but there are also much worse alternatives. The system we have really is an imperfect attempt in need of adjustment and not a deliberate attempt at cruelty. The cruelty is a bug not the feature.