You are right but your logic works in tcp connections, that why the method named "udp hole punching", in udp there is no handshake that means i can send you a udp packet looks like it's coming from S_IP:S_PORT.