Not only that, but letting all traffic through Varnish is not a good utilization of its resources at all.
The question is, how could I set up multiple servers on the same port? Or what else do I need to put in front of Varnish to route WebSocket connections elsewhere?
I'm starting to think this use case is not easy to solve under one IP address.