There's no way to do that without tying the voter to the vote somehow. Why not hash the voter ID + a secret of your choosing and store that with the vote? Then, even if counting by hand, you can verify your vote was indeed cast for X candidate.
I would rather be able to verify that my vote was cast to the desired candidate than to have a strictly anonymous (but unverifiable) voting system.
EDIT: Mail in voting is skimmed with little to no risk. Oops, lost 1% of that zip code...
Assume a million votes are cast in an election. To move the needle 1%, you need 10,000 votes. And let's assume a single fake voter can realistically cast ten votes in different precincts, considering vote time and travel time (they can't just keep coming to the same precinct, for fear of recognition).
So you need 10,000 fake registrations, that must match real addresses, and you need 1000 people working all day at fake voting. One voter out of every thousand would be working for this conspiracy.
Now, you need to do this in complete secrecy. Nobody can talk. Nobody can gather evidence. They can't go to the police, or the media. Even one leak is enough to not only ruin the scheme, but create a nationally known scandal. And if it can be connected in any way to a particular political party, that party's name would be dragged through the mud across the country.
It's insanity. No one could safely pull it off.
You know what can be pulled off, though? Voter suppression. Make it much more difficult for certain categories of voters to actually vote. Such as poor people who might not have ID (or might hesitate to use it). Then, convince a large swath of the public - people like you - that this is about security, not suppression. That form of election manipulation is now seen as patriotism, not corruption.
I can see that happening maybe a few times, but it seems ineffective on a large scale.
If they already voted? "Oops, I forgot."
If they haven't voted and come in later, so what? You're long gone.
This may sound laborious and slow but it should be enough to block repeat voting, IMHO.