The difference between having a passport or a national-issued digital identity card that has a private key inside is not much different. In passports, there are different security mechanisms in place that make it hard for an adversary to fake them. For digital identity cards, similar mechanisms exist so that their issuance does not depend on a single entity. The full list for which the identity cards are issued can be audited by sampling and finding follow-ups to see if the issued identity is owned by the claimed legitimate person and is independent, whether the thing is a passport or an identity card. Although, I don't know whether such checks are actually being done in practice. The security of digital identity is being assured with an increasing variety of documents that can be signed digitally using such government-issued cards.
Afterwards, when we have a list of public keys that are eligible to participate in the vote, E2E verifiable evidence prevents any involved party or coalition from deceiving the public with a manipulated election tally, as that would not produce valid cryptographic proof.