A couple of States even allow inmates to vote while incarcerated. In my now-home State, inmates are encouraged to vote and sometimes politicians will even go campaign inside the prison. That's pretty rare, but has happened.
So, it's a temporary loss in most cases. Notably, Maine allows inmates to vote and Maine is overwhelmingly white. Vermont is the same way. In my quest for more information, I found some commentary about that aspect.