Someone would need your 256-bit key to do anything to an exposed Wireguard port.
In the same theory, someone would need your EC SSH key to do anything with an exposed SSH port.
Practice is a separate question.
It's slow to scan due to ICMP ratelimiting, but you can parallelize.
(Sure, you can disable / firewall drop that ICMP error… but then you can do the same thing with TCP RSTs.)
If you're blanket dropping all ICMP errors, you're breaking PMTUD. There's a special place reserved in hell for that.
(And if you're firewalling your ICMP, why aren't you firewalling TCP?)
Wireguard is explicitly designed to not allow unauthenticated users to do anything, whereas SSH is explicitly designed to allow unauthenticated users to do a whole lot of things.
I'm sorry, what?