The serial number in my scheme would not be linked to voter. It's not registered, only signed for the validation code.
With that, we are left with the following attack vectors: the server and its software, either via hacking or via subtle rule tweaks, targeted ballot invalidation, voter pressure. As a technopesimist, I'm especially uncomfortable that a key piece of the process is an opaque blob of silicon that can't meaningfully be inspected by a human observer. Echoes of Diebold voting machines, plus billions of dollars poured into elections. But I can see why HN audience is prone to be persuaded this is a good idea.
Clarification: serial number is mailed with the ballot, contains a signature (like two part keys for API f.ex.). You submit the serial for signing through authentication mechanism (verifying the voter). The signature can be either PKI or hash. This way you can validate serials, signatures and have them independent from the ballot after separation. If you have designated drop-off locations you insure the ballots are tamper-proof after being filled out (barring massive system-wide fraud).
http://mediad.publicbroadcasting.net/p/northwestnews/files/s...