I don't follow. I'm sending (or trying to send) messages to my contacts. If I know their phone number, I'm going to try to initiate a Signal conversation with them. So I ask the Signal server for a signed prekey. Your argument is that Signal should not respond with "I don't know this person" and should instead respond with something indistinguishable from a "real" response. So they must send me something that looks like a signed prekey, right? Well then I would use that in order to do a key exchange and now we're in the situation I described above.