OK, just to cite one example: The population of North Carolina as of 2012 was 9,752,073. 23.7% are under 18 and therefore ineligible to vote. That leaves approximately 7,440,831 eligible voters[1] (I'm ignoring convicted felons and other edge cases for now).
According to the State Board of Elections, there are currently 6,438,531 registered voters. 2,764,322 Democrats, 1,984,251 Republicans, 20,901 Libertarians and 1,669,057 unaffiliated.[2] So that gives us around 1,002,300 potential voters who aren't even registered. So around 13% of the eligible voters aren't even registered and obviously don't vote.
Now that's just one state, and that doesn't even attempt to account for people who are registered, but don't bother to vote, or vote infrequently. But it does suggest that there's a not-insignificant number of people who are sitting on the sidelines during elections.
Edit: Also, as one might expect, there's a Wikipedia page on this topic:
http://en.wikipedia.org/wiki/Voter_turnout_in_the_United_Sta...
http://felonvoting.procon.org/view.resource.php?resourceID=0...
Your North Carolina example still stands, as only (only being relative here) ~80,000 people are disenfranchised due to felony status. But, in states like Florida, Mississippi and Kentucky, it can be as high as 10% of the voting age population.
I bet most of them are people who don't care about politics. Good luck trying to engage them, people have been trying to do so for decades and have failed. The simple reality is that life is comfortable enough for most. Few are starving, few are persecuted- if you're earning enough to live a relatively comfortable life, why bother voting?