Go by Example: Timers and Tickers
mmcgrana.github.com
mmcgrana.github.com
I ask out of curiosity: are there other statically typed languages where you can do stuff like this? Java has Futures, but those are far more heavyweight. Clojure has concurrency constructs which are pretty close, but Clojure is dynamic. I only know a little bit of Haskell, so there might be something like this in there.
Also I'm sure there's about a dozen C++ libraries that will do it.
I want to read data from a remote port every second, with a 0.1s timeout. Any longer and I can treat it as a failure. If I put ReadRemotePort() in a forever loop with Sleep(1 second), even though the timeout will cause a failure as expected in 0.1s, there will be go func() threads still active until net.Dial has timed out. So at any given time, I could have 180 go func() calls just waiting for net.Dial to timeout at 3 minutes. I still haven't found a way to avoid this.
Q. What are the limits to scalability with building a system with many goroutines?
The primary limit is the memory for the goroutines. Each goroutine starts with a 4kB stack and a little more per-goroutine data, so the overhead is between 4kB and 5kB. That means on this laptop I can easily run 100,000 goroutines, in 500 MB of memory, but a million goroutines is probably too much.
For a lot of simple goroutines, the 4 kB stack is probably more than necessary. If we worked on getting that down we might be able to handle even more goroutines. But remember that this is in contrast to C threads, where 64 kB is a tiny stack and 1-4MB is more common.
Yeah not really. C threads normally have a fixed-size contiguous virtual memory space allocated to them, but the space they don't use is not actually consumed. You can even call madvise() to release memory on returning from a deep stack so you only actually "use" 4k of memory afterwards. The only significant resource consumed is virtual address space, which is effectively unlimited on 64-bit systems (why design a new language around 32-bits systems?). For that matter "C threads" could use segmented stacks as well if they wanted to.
But "goroutines" also have to be concerned with the total stack space they may use. For example, suppose those 100,000 goroutines were network servers and a hostile client could manipulate the connection to block the routine while it was consuming 400 KiB of stack. Now an attacker (or random confluence of events) can cause your program that normally averages 1% memory to use 100x as much as normal and run out of memory.
Space is almost never a reason to use coroutines over threads.
hmm so you're saying a request causes 400KB to be allocated, and an attacker hits your server with 100k requests designed to cause your server to stall? The same criticism could be leveled against... any system where a requestor can allocate memory and stall a handler that is independent of the incoming request loop. How would this be different in a different concurrency model?
>Space is almost never a reason to use coroutines over threads.
I wouldn't necessarily say so. I have a polling service that uses a few thousand standing goroutines. The memory overhead for each individual goroutine is so low that I can just spawn them casually. Each goroutine only actually does work for a few seconds every few minutes; the vast majority of the time, each goroutine is spent sleeping. For this type of problem, I find that organizing things with goroutines and channels makes my code very easy to reason about. That, to me, is the biggest win of Go's concurrency model; that it's very easy to reason about.
Absolutely agree. I just wish the net.Dial calls would let me specify a timeout and wouldn't hang around for 3mins in case something is down.
That was actually the point. You have the same space considerations. Being coroutine instead of native thread is just a small constant factor... if this factor is important then you are almost certainly doing it wrong.
// TODO(bradfitz): the timeout should be pushed down into the
// net package's event loop, so on timeout to dead hosts we
// don't have a goroutine sticking around for the default of
// ~3 minutes.See the comment here: http://golang.org/src/pkg/net/dial.go?s=3003:3074#L110
Edit: In practice, this isn't really a problem at all unless you're calling Dial something like a hundred or thousand times a minute and they're all timing out. Goroutines are very cheap. Still, it is an issue and will hopefully be resolved in Go1.1: http://code.google.com/p/go/issues/detail?id=2631