That would enable people to prove what they have voted, which in turn enables vote-selling and/or for the head of the family to force other family members to vote a certain way.
Not enough voters would handle their keys securely. Maybe if every voter was given some kind of hardware which could sign their votes offline. That's a big, risky, upfront expense, though.
Still wouldn't work probably -- think of how many times the average person misplaces their car keys in a given year. Now give them a piece of hardware, that they're only going to use every 2 years (which is a stretch given Americans' voting patterns, so realistically it's every 4 years if ever), and tell them to (1) keep it safe and (2) remember where they put it.