Forgive me, as I haven't used signal, but I don't see how whether they are sitting next to you or not changes the problem.

If I can generate a key that hashes to the same value as your key, I can convince anyone I am you. If I can generate a second collision for a third party's key, I can convince you you are talking to that third party, as well. Generating hash collisions is, as I understand it, pretty well modelled with the birthday paradox (and variations like the one I linked). Physical proximity seems entirely unrelated.