Golang custom transports and timeouts

biasedbit.com

77 points·ovokinder··6 comments

I had a similar (though unrelated) problem early on with Golang writing a port scanner, which forced me to hoist my own event loop (build around an explicit select call) into my program: timeout or not, I needed lots and lots of OS file descriptors, and by relying on Golang's own event loop to close stale ones on timeouts, I'd invariably run out.

Similar unrelated problem here, too, on the sole machine now running the Heartbleed test.

After recompiling with Go 1.4 (from Go 1.2 - apparently the network poller was rewritten?) I'd start running out of file descriptors really fast, all leaking deep in net/http.ReadRequest.

Turns out http.DefaultServer doesn't have any default ReadTimeout, so when a client goes away before finishing a request I'd lose the goroutine and the file descriptor with it.

So yeah, you probably also want to explicitly set http.Server.ReadTimeout if you are not doing weird streaming/"comet" stuff.

That's related to my problem, but not the same; what I needed was control over when an fd/goroutine would be destroyed. Explicit timeouts (at least in 1.0, but I think still) couldn't give me that. I needed an obscene number of fds, though.

> I needed lots and lots of OS file descriptors

Why?

I ask because I wrote a port scanner professionally once, and I got away with using one fd for network I/O. I can only think of one case where you end up using a lot of fds and that is if you're using TCP sockets and connect(2) where you'd end up with one fd/tested port, and that's not good.

It had to work unprivileged in userland.

If you're going across the network, a slow timeout is often trickier than an explicit failure.

I wrote about handling this in Python: http://www.mobify.com/blog/http-requests-are-hard/

Reply on news.ycombinator.com