It's not about using the system as an "user". It's about the fact that it bases democracy on principles which are way beyond the mathematical capabilities of 99%+ of the electorate!

I mean, if my government decides to use some crypto/blockhain magic, there is no way I can figure out if using hash algorithm H in a method X is valid choice or if it is a backdoor/negligence.

This is an issue which simply does not happen with paper ballots (backdooring these is of course possible, but really expensive especially at scale). The attack surface is easy to reason out, unlike with cryptography.

Maybe it does not matter though (and maybe I am afraid of mathematicians secretly taking over the world, ha!)

Accounting uses double entry (credits & debits), election administration does not. Compulsory voting could make election administration look more like accounting.

Blockchains explicitly remove privacy, whereas voting should be the Australian Ballot, aka private voting & public counting.

You only care that the ATM is fair to you. An election system needs to be fair to everyone else.