The parties cannot obtain the shared secret the usual way, the Diffie-Helman exchange. It cannot be performed, because it requires back-and-forth communication, which we are trying to establish in the first place. Of course, we'd still need to use certificates signed by public authorities to deal with the standard MitM concerns.
The alternative would be for the sender to use the recipient's public key to encrypt the source IP address. That would require a DNS-like system that would store and serve public keys corresponding to a given IP address you are trying to communicate with. This is something doable, especially in IPv6 world where you don't really have to reuse IPv6 addresses, but it leads to all kinds of practical problems, figuring out which is probably best left as an exercise for the reader.