But it's also public info, right! What if someone else just walks up and gives my info??
Then I won't be able to vote. I'll have to file a provisional ballot and a complaint that someone stole my vote. These sorts of things are tracked and it turns out it's very rare.
As a means of altering an election outcome, it just does not scale. Even to fraudulently elect a town dog catcher, you would need hundreds of people to fill out fraudulent ballots, one by one by one. Where are you going to get those people from? Why would they spend their time doing that? What's in it for them? What incentive would that have to keep it a secret?
Can you see why it is dumb to worry about this kind of vote fraud?