This sounds great until we realize that the vast majority of crimes, even minor ones, have someone on the other side of them.

Simple assault, theft, fraud, reckless driving. Hey, someone came up and beat the crap out of you for the $20 in your wallet? Well sorry, we can't do anything but talk about how they might of come to this point. Someone who gets the flu, unless they're acting to intentionally infect others, has no "victim".

That doesn't mean justice should be based on retribution, but removal of the offender from society is something which society typically requires.