Electronic voting significantly lowers the barrier for commiting fraud by election administrators.
The difference between having a passport or a national-issued digital identity card that has a private key inside is not much different. In passports, there are different security mechanisms in place that make it hard for an adversary to fake them. For digital identity cards, similar mechanisms exist so that their issuance does not depend on a single entity. The full list for which the identity cards are issued can be audited by sampling and finding follow-ups to see if the issued identity is owned by the claimed legitimate person and is independent, whether the thing is a passport or an identity card. Although, I don't know whether such checks are actually being done in practice. The security of digital identity is being assured with an increasing variety of documents that can be signed digitally using such government-issued cards.
Afterwards, when we have a list of public keys that are eligible to participate in the vote, E2E verifiable evidence prevents any involved party or coalition from deceiving the public with a manipulated election tally, as that would not produce valid cryptographic proof.
And even if you could, you see a name you never heard of, so how do you check whether it is a real person or not? Without having access to government databases.
Trusted third-party auditors can have such access without linking the public key to a person's identity. They can also access government databases on a sample basis. This is no different than linking the passport serial number to the person's identity.