* Lossless UDP? Is there a reason not to do TCP?
* There is no way to know if the public key is genuine, so the system is very sensitive to MITM.
* The key exchange is inadequate. Why not do DH if it's just to have session keys?
* The system is very easy to brute force as the acknowledgement is based on a known plain text. This is very bad.
A quick glance at https://github.com/irungentoo/ProjectTox-Core/blob/master/co...
I found a potential buffer overflow at line 143. If an attacker sends a large file, what happens?
Making crypto software is not just a question of wrapping a crypo lib (in that case NaCl) with a GUI. There are some tricky security issues as how you use the crypto.