I found that it corrupts the FD by using this call:
https://github.com/rcrowley/goagain/blob/master/goagain.go#L...
Not sure why it happens though, but it led to all sorts of strange intermittent issues with broken connections.
Once I replaced this logic with passing files using GetFile().Fd() instead it started working fine, so beware of this. I still wonder why it happens though.