That is not true. See work by Andrew Neff on verifiable shuffles of secrets. It is practical to produce a proof that can be verified by the voter but not useful in proving which way the ballot has been cast to anyone else.

There is a simple solution: generate a random public key as part of the voting process and print it out for later verification but also include plenty of other people's public keys in the printout.

The voter can remember the position of their own key if they choose to for later validation against the open ledger and can use the others for deniability (if needed).