Goroutines, Nonblocking I/O, and Memory Usage
eklitzke.org
eklitzke.org
That also means reads aren't interruptible (unless you count closing the descriptor as interrupting, which is a blunt hammer indeed). AFAIK the only way to do this is to set SetReadDeadline() on the connection to a smallish value, then let the interrupter block on a channel that's invoked when the deadline is reached.
Go's concurrency model is pretty friendly, but it's also unfriendly in many surprising ways. (Don't get me started on the whole nil channel thing, or the problem of channel closure/ownership, or channel performance overall.) It doesn't surprise me the least (while it did surprise the Go team) that Go fell flat among the C/C++ system programmer demographic.
Anyone want to chip in about how Rust, Swift and Nim compare here (regarding the article)?
They try to keep it simple. As with all I/O: it's blocking your current goroutine, so if you want it non-blocking you spawn a new goroutine to do this in, pass it a channel, the spawned routine proxies the result to it and your main routine does select on that.
The semantics for reading sockets are different from reading channels (compare the types; one is multi valued, the other single valued---EDIT: no they're not, exactly, see [1]), so it kind of makes sense you can't share the same construct.
I'm not a Go apologist by any means, I think it has a tremendous amount of flaws. But I/O, (a)synchronicity and concurrency are not among them. They are simple and consistent, and (most importantly) orthogonal building blocks which you can combine at your leisure to create more complex constructs.
(Besides, how do you interrupt a read, anyway? Or any syscall? You can't do that in C, either...?)
[1] edit: fair is fair, go channel reads are (of course) also multi valued. however, the types still don't exactly match, since you can't pass arguments (buffer) to a channel read. you'd need some extra hoops to jump through, as other commenters suggest, which starts becoming magic and breaking down the simplicity and orthogonality of the constructs.
The usual trick on POSIX is to use a self-pipe, since select()/epoll() only work on file descriptors and not, say, pthread primitives. Windows is slightly better here: WaitForMultipleObjects supports waiting on any primitive that supports blocking, so you wait on the socket + on an event object, and signaling the event object will unblock the wait. Windows also has completion ports, and the APIs are generally more extensive (not a fan of Windows, but they got this part very right).
I do understand why Go is the way it is. But it's important to recognize that it's a compromise in design that inherently reduces your flexibility — such as in this particular case. I don't think there's anything technically preventing Go from supporting non-blocking I/O, though.
Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? One major difference is that reads can fail with an error, and channel reads can't.
Edit: Also, I didn't mean that select{} on a file descriptor ought to actually return data. It would be more appropriate for it to yield a true/false value to signal that the FD has data, so you'd do "case <-fd: fd.Read(bufSize)".
> Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that?
I was trying to preempt pedantry, but ended up being the pedant myself :P "You can simulate multi valued responses in a channel by passing a struct."
Hopefully not to derail this too much, but what do you see as the major flaws?
I'm asking out of curiosity - not trying to start a flame war.
I care about collecting goroutines, and would advocate that everyone should care about this. There's been a litany [1] of blogs about this recently, if anyone doesn't consider this self-evident. It's just systematically sane. Letting goroutines spin off into the distance with no supervision is extremely likely to represent either a resource leak, a missing error handle path, or an outright logic bug.
In a dream world, we might have Erlang-style supervisor trees so this can all be done with less boilerplate. It's almost even possible to do this as a library in go! ... except for this blocking IO problem, which throws a wrench in things: any supervisor of a goroutine that may be IO-blocked may itself be indefinitely stuck, and so on up-tree, making the concept unworkable.
---
[1] Some recent articles hammering the point of "collect your goroutines":
- https://dave.cheney.net/2016/12/22/never-start-a-goroutine-w...
It takes the Wait-for-children behavior path by default, and that seems much safer to me. The downside is, as we're describing: if you do get stuck in IO somewhere, you're a pretty unhappy camper. This library makes the choice of (by default; it's a handler func you can override) logging warnings to stderr if a task doesn't exit within 2 seconds of being told to quit... but this still feels like something of a hack; more of a debug measure than a concrete way to improve system stability. Better ideas would be highly welcome!
With Tokio, you write non-blocking IO stuff as "futures", similarly to promises in JS. These then get compiled down to what's effectively a state machine, and they run in an event loop on top of epoll/kqueue. That event loop is single threaded, and fully non-blocking.
I am not a super, super expert on these low-level details, but I believe that this should mean that it is closer to the C code than the Go code.
I like futures, though cancelation has historically been a controversial topic there, too (at least in the JavaScript world; JS promises are not cancelable, which is problematic for HTTP requests [1]).
I do like that you guys chose to not make green threads a language-level construct (as Go did with "go" and goroutines), which means you have the option of supporting multiple models, for the programmer to choose according to their particular use case.
Yup! The danger there is that the ecosystem may fragment, however, basically everyone is rallying around tokio, including the other people who were working on similar libraries, so in practice, this should be fine.
* Async await
* The async dispatcher (event loop)
* Selectors module
Each builds on top of the layers below it. The selectors module is an abstraction for the various system `select()` mechanism such as epoll or kqueue. If you wish you can use it directly, and some people have done so successfully[1]. But you must be aware that on Windows it only supports `select()`.
The async dispatcher implements the proactor pattern and also has an implementation of IOCP. It implements the proactor pattern for other OS' via the selectors module. Futures are also implemented here, and indeed all the procedures return a future.
The final layer is the async await transformation. This allows you to write asynchronous code in a synchronous style and works similar to C#'s await. The transformation works by converting any async procedures into an iterator.
This is also single threaded and fully non-blocking and does give you a lot of control. In theory you should be able to work with any of these layers interchangeably (as long as you only care about non-Windows platforms).
https://godoc.org/golang.org/x/sys
https://godoc.org/golang.org/x/sys/unix#EpollWait
https://golang.org/pkg/syscall/
Of course it is a bit of a pain to write OS specific variants if you are building a completely crossplatform solution, but it is supported out of the box:
https://dave.cheney.net/2013/10/12/how-to-use-conditional-co...
That said, I agree [regarding the] puzzling decision to not to treat 'select' as a general contract for all asynchronous ops as that fits perfectly with the CSP approach.
This would also simplify most of the code that tries to pull events from different sources, including I/O. Right now each I/O needs to be extracted in it's own goroutine (adding error handling makes it worse):
fromIO := make(chan []byte)
go func() {
for {
buf := myPool.Get()
myIO.Read(buf)
fromIO <- buf
}
}()
for {
select {
case a := <-fromIO:
// do one thing
case b := <-fromOtherChan:
// something else
}
}
for {
select {
case <-myIO.ReadReady:
buf := myPool.Get()
myIO.Read(a)
case b := <-myOtherChan:
// ...
}Also some would say POSIX is the language runtime that ANSI didn't want to approve.
Aren't libc etc. passive and effectively just libraries?
Having select{} work on io.Reader and io.Writer would be great!
One of the beautiful things about Go is the designers carefully chose to confine language magic to only a few fundamental areas (channels, goroutines, arguably maps) and resisted going beyond that. Go approximates "C plus Concurrency and Better Data Structures" and nothing more.
"Extra", yes, "magic", debatable. (Which, to be clear, I mean straight, not as a rhetorically-weakened pure disagreement.) There is precedent for using the io.Reader and io.Writer interfaces but allowing structs that implement those interfaces to implement additional interfaces to gain additional capabilities already, especially in the io.Copy function: https://golang.org/pkg/io/#Copy (Click on the Copy name to pass through to the underlying implementation, which bounces through to another function which should still be on your screen, and the first few lines show the special casing.) Though I also find myself with some frequency having to accept a Reader or a Writer and examining it to see if it's also an io.Closer, because if I want to wrap a new io.Reader around an underlying io.Reader for some reason, something I do quite a lot, it's important to propagate proper closing behavior, especially when wrapping with gzip or something else that really needs to be closed and not just sort of trail off at an arbitrary point.
Making io.Reader "just work" in a switch statement would be magic, but using a similar technique to the above it might be possible to offer a new "io.ReadableChan(io.Reader) <-chan struct{}" (or the obvious WritableChan extension) function that files or sockets would implement using a polling mechanism, and everything else would get an automatic new goroutine created that used just io.Reader's existing interface. This could offer a zero-buffer-until-IO implementation without having to modify io.Reader. The resulting channel coming back from ReadableChan might have to be a new sort of magic channel internally, but that would be an internal detail that shouldn't ever rise up to your code.
Alternatively, it might be a "io.ReadableChan(func() []byte) <-chan []byte", waiting internally until it can be read then determining the []byte to use for the read by invoking the provided function. I'm just spitballing here, not preparing a change proposal. (For such a thing I'd also want to consider whether that should be "func () ([]byte, error)", for instance.)
This is one of those places where Go favors practicality over purity, and is very much not channeling Haskell. Checking whether an interface value implements another interface is probably something you shouldn't be doing all the time, but it's not inherently un-Go-like or anything. You're just taking more responsibility. Much like how you are supposed to share memory by communicating but Go will still let you communicate by sharing memory if you want.
The compiler actually just transforms everything channel related into function calls within the runtime package. The majority (probably > 99%) of the implementation of channels and select is Go itself.
E.g., see: https://golang.org/src/runtime/chan.go
That this pattern exists and it would be nice if there was a solution, I don't really care about the implementation.
Even if a socket's Selectable implementation just abstracted away the goroutine from your first example, it would make the developer experience a lot nicer for me.
The fsnotify lib at https://github.com/fsnotify/fsnotify has a similar challenge at a high level (ferry events from various syscalls into a Go chan) and may be interesting as a reference.
But: perspective. If KBs per idle conn are eating a well-provisioned modern server box whole, then back of envelope you might be at, like, hundreds of thousands to a million conns for a single box (few KB/conn * 1M = few GB), and, like, huge congrats for getting that far! Not the absolute limit of the hardware, maybe, but clearly not a toy!
Other folks also note that once you're juggling that many connections there are other things that could become big factors in scalability, like whether your kernel and network will remain happy, can you keep up if more conns than expected suddenly become active, and (I'd add) will the rest of your code (the stuff besides reads/writes) be able to keep up.
Still, if anyone wants to wrap the epoll-results-to-a-chan thing in a package, they can always post something + see if anyone uses it to do cool stuff. As you suggested, it's plausible there are places it can make code cleaner.
But you're right that myIO.ReadReady doesn't give the guarantee that there is enough data to fill the len(buf) on the next Read call, which would make it then blocking.
I think the author says something similar put differently in the first paragraph.
Something like rakoo's trick is interesting, with the caveat zzzcpan added that it takes at least a tiny buf, not zero-len, but if you don't actually hit this wall I'd charge forward the regular way.
The readyness model avoids this, as described in the article. But it will also lead to a more event driven programming model.
I think the preallocation might be a problem if the use case is a server that holds an enormous amounts of mostly inactive connections (e.g. websocket server). For most servers I don't see a problem with it.
This method also has a benefit for performance: You often can totally avoid dynamic memory allocation during runtime with this model. You allocate the receive buffer once for the connection and then reuse it for the whole lifetime. In the push model you either need to allocate a fresh buffer for the new data or retrieve one from a pool, which is still more expensive.
There is a security concern here; you'd better be sure to flush the buffers correctly or you could cause memory to leak between sessions.
If I were writing a server to handle a sensitive task such as executing payment transactions I'd probably take the less memory efficient approach of allocating a new buffer for every connection to lower the risk of data spilling between connections.
_, err := conn.Read([]byte{})
if err != io.EOF {
return err
}
// we know there is something to read
buf := pool.Get().([]byte)
n, err := conn.Read(buf)
// process n, err and buf as needed
// if there is more to read, you may need to loop over conn.Read
// after some inactivity timeout
buf = buf[:0]
pool.Put(buf)Won't matter much though, as the memory usage of a single goroutine is quite significant and doubling it by having a buffer preallocated is not something to care about.
2K as of 1.4 (was 4K before 1.2, 8K in 1.2 and 1.3).
But yeah it's a good point, the intrinsic memory overhead of goroutines means even if buffer pooling worked the memory use of the system would still mostly follow the "naive" estimate.
You may want to message the author to remind them of that, they may not have thought of that concern.
I suppose the intent is to then use that 95% memory savings for other work.
So what's the point of making this optimization? The system is going to be hugged to death if the number of client connections approaches 100%, because the system will not have enough available memory.
If 100% client connections is causing memory problems, rather than keeping a pool of buffers so that inactive client connections have less footprint, it seems like a better solution is to decrease the number of available client connections. Then, provision more servers or more memory if more client connections are required.
Since HTTP connections are typically kept alive between requests, it's very common for them to be idle. It could be the case that 99.9% of the time, X bytes of memory is enough to avoid transfer delays, but to cover the 100% case, you'd need 10X that.
I think it's reasonable to decide that avoiding delays in the 100% case is not worth 10X the cost in memory.
If you use a magnitude fewer resources, you can combine more services on one machine, ultimately allowing faster servicing and a better user experience.
And as I understand that's actually how many Go programmers think. If you don't care about resources, you can set up a simple server in Python. The beauty about Go is that it gives a good developer experience while still allowing to write efficient, optimized programs.
(TCPConn implements ReaderFrom, but not WriterTo)
Yes I understand the purpose — hence suggesting WriterTo which is supposed to do that — I simply noted that conn.Read(w Writer) would make Conn not extend io.Reader anymore since it takes a buffer.
You will also need to buffer incoming data from a stream protocol like TCP to combine consecutive reads into complete application layer packets.
A better approach can be seen with Registered I/O on Windows, which lets user-mode programs register buffers with the kernel so that they can be locked for the NIC to DMA into: https://technet.microsoft.com/en-us/library/hh997032(v=ws.11...
In the classic C10k scenario, most sockets don't have a segment in flight at any point in time, so their buffers will be empty. It would be a huge waste if they were actually occupying memory.
One could imagine a function that takes a set of net.Conns and returns the ones that are available for reading. You'd then pass those off to a goroutine for processing with blocking I/O as usual.
One way go can make it nonblocking is to do the read/write and return the result immediately, but this kind of call requires a call back function to be provided, so instead of parking/unparking the goroutine, the call back function is called (either directly or by "go cb_function"), It can be tricky to make it work, since the initial goroutine might not be allowed to call read on FD anymore (maybe exit from the goroutine if the read was unsuccessful). In addition, you still need to provide a buffer to read from, the 1 byte trick works though, but the buffer management can be done either if the read was successful or in the call back function. Also this approach makes the programming asynchronous and similar to event programming.