That's one of the original design requirements for wireguard. Unless a packet is signed with the correct key, it won't respond at all.