The trick is this: in a traditional DH based key exchange, you need A) your private key and B) the other person's public key. Let's say your keypair is a/A and my key is b/B (the lowercase identifier is the private key, the uppercase one is the public key).

To do an exchange, you send me A, and I send you B. You compute the DH exchange with B and a, and I compute it with A and b. The important property of the DH function is that computing DH(B,a) is the same thing as computing DH(A,b).

Now, if I steal your secret key 'a', I can do two things. One, I can obviously impersonate you. But two, I can impersonate anyone else, because I can calculate the shared secret. The reason for this is that I need your private key, but I only need the other parties public key.

Let's say you try to communicate with me now and someone has stolen 'a'. They can already have my public key, B, because it is public. Thus, when the exchange happens, you send me A. The attacker knows the secret key, 'a', already. So the attacker can intercept the communication, calculate the DH exchange of 'a' (which they stole) and 'B' (my public key), and they have calculated the shared secret. It is not possible for you to tell you have exchanged with the attacker, and not me. They never need my private key, only my public key, which will come through during the handshake. So it can always be intercepted. Remember, DH(B,a) and DH(A,b) are the same, so if both sides calculate DH(B,a), that's completely legitimate.

This is a very simplified view (and I haven't looked at the bug report in detail, TBQH, so the case for Tox is probably slightly different), but it explains why if your keys are stolen, anyone can be impersonated to you: because the impersonator only needs public information (public keys) from that point on to forge any exchange.