I see you plan to publish the source -- do you have a way that someone can verify what is running on their device, such as using common components so they can load the code themselves, or maybe a version that runs as installed software on a desktop computer? (This wouldn't be as convenient, but it could provide safety to the ecosystem if it could detect hostile clients.)
EDIT I wonder how much computational power it would take for an attacker to do a man-in-the-middle attack that recognizes each side saying "the code is 123" and change the voice to say "the code is 456."