Not unless you want to get through firewalls seamlessly. UDP "state" tracking is a thing, and if my firewall sees a UDP packet destined for it without knowing the remote IP address it's going to drop it.
The load balancer will route any ongoing sessions with the session id, and not the ip address.
Also, most firewalls on end-point devices (think built-in to the OS) are stateful firewalls.
Granted, it requires transferring of ICE connection candidates out of band (via the horrors of SDP)