Poor choice of wording on my end, I was coming up with the idea on the fly. I'm sure someone else can come up with something better with a bit more thinking :)
The association between vote keys and personal info does not need to exist. ID keys have to be issued by the gov but the vote signature keys can be derived from them and/or signed by a 3rd party like an NVO or a combination of govs/NVOs.