What do you do when duplicate serial numbers start showing up? I'm assuming you won't know who was issued which serial number, and if it's truly randomized you wouldn't even know where they were sent.
The chaos that would ensue from this is staggering.
And your proposed resolution means someone could DoS an election by copying their ballot and submitting it.