Of course, the potential for violence from other inmates is always there, but I suspect that is not a serious problem in most prisons. And the potential for violence from a random citizen in the street is relatively high in poorer areas.
Overall, prison may be a better alternative to severe urban poverty. This may be another reason why the poor are much more likely to face incarceration. Not only does their plight cause them to suffer a lack of alternatives, but the punishment for crime doesn't look so bad to them.