In the case where disproportionately more poor people are of a certain race then it can be seen as racist (as it affects the population of that race differently). If the reason that disproportionately more poor people are of a certain race is because of racism, then a policy that disenfranchised the poor would effectively extend economic discrimination into political discrimination.
Though I tend to think that even if we remove the economic effects of racism such that disenfranchising the poor couldn't be called racist, they would still be classist and should be avoided where possible.