The race condition is definitely there, but if you use t.Cleanup to close the listener then the listener does not stop listening on the port until the end of the test. If you try to bind an http server to the port during a test then your http server will stop and return the error "bind: address already in use".
I'd be interested to know how you're working around that, if you are able to share.