I used to think the same thing, until I read an article that made me look into it. The article I read was a local affair that pointed out that inmates are encouraged (in my state, Maine) to file absentee ballots in the town they lived in prior to incarceration.

There are just a couple of states that allow that, but I learned that quite a few jurisdictions restore rights after incarceration, after probation, etc...