I have not looked deeply at the implementation, at all, but given that the actual site refers to this as an "algorithm" for LVS, I kinda suspect all the heavy lifting is happening at the kernel level within LVS, and the go code is merely acting as a control program that tells the backend how to behave. Many high performance systems have lower performance scripting languages or similar built in or bolted on, in order to provide easier means of development without impeding performance significantly. LVS is no exception.
The LVS-based systems I've built and used in the past often had a number of non-C components (mostly Perl, as it was 10+ years ago), for health checks, data gathering, making balancing decisions, etc. It is entirely possible this is the kind of work Go is doing (again, I haven't looked deeply into the code...but, I don't immediately see anything indicating Go is doing the actual load balancing, but it is obviously doing health checks and providing management access).
Finally, Erlang is a garbage collected language, and yet it has been used for a couple of decades for this kind of workload. So, evidence strongly suggests it is possible for a GC language to do things like this (though, again, I don't think Go is doing the network layer work here, since LVS is in the picture).