In the video, Ron Rivest specifically addresses how to allow both an anonymous vote with (some) mutually exclusive end-to-end verification.

I completely agree that electronic voting is currently a terrible idea. The other talk I linked to by Andrew Appel strongly encourages using only paper votes and statistically verifying them by manually recounting a random sample of voting locations.

However, I'm willing to listen to Ron (the "R" in "RSA") Rivest's ideas about using encryption in voting. I think the idea needs more work, but it's interesting approach that does attempt to address both anonymity and verifiability.