Restricting origins doesn’t work thanks to a fun little trick called DNS rebinding. Let’s say you get the user to browse your site, say, attacker.com. They lookup your DNS and see that it’s 1.2.3.4, with a TTL of (say) 5 seconds.
Five seconds later, the user’s DNS cache entry expires. Your page triggers a new request to attacker.com. Since the TTL has expired, they do another DNS lookup - and you serve them a different IP address, say, 192.168.1.1. Now they make a new request to that IP believing that they’re still accessing attacker.com.
With HTTP requests there’s at least the possibility of detecting these attacks via the Host header (even if that doesn’t usually happen in practice). With raw UDP, however, the attacking packets would be indistinguishable from legitimate ones.