Hacker News Reader

> You have to read from it, and deal with the result.

They try to keep it simple. As with all I/O: it's blocking your current goroutine, so if you want it non-blocking you spawn a new goroutine to do this in, pass it a channel, the spawned routine proxies the result to it and your main routine does select on that.

The semantics for reading sockets are different from reading channels (compare the types; one is multi valued, the other single valued---EDIT: no they're not, exactly, see [1]), so it kind of makes sense you can't share the same construct.

I'm not a Go apologist by any means, I think it has a tremendous amount of flaws. But I/O, (a)synchronicity and concurrency are not among them. They are simple and consistent, and (most importantly) orthogonal building blocks which you can combine at your leisure to create more complex constructs.

(Besides, how do you interrupt a read, anyway? Or any syscall? You can't do that in C, either...?)

[1] edit: fair is fair, go channel reads are (of course) also multi valued. however, the types still don't exactly match, since you can't pass arguments (buffer) to a channel read. you'd need some extra hoops to jump through, as other commenters suggest, which starts becoming magic and breaking down the simplicity and orthogonality of the constructs.

Sorry for being unclear. You don't interrupt the read as such. You interrupt the select.

The usual trick on POSIX is to use a self-pipe, since select()/epoll() only work on file descriptors and not, say, pthread primitives. Windows is slightly better here: WaitForMultipleObjects supports waiting on any primitive that supports blocking, so you wait on the socket + on an event object, and signaling the event object will unblock the wait. Windows also has completion ports, and the APIs are generally more extensive (not a fan of Windows, but they got this part very right).

I do understand why Go is the way it is. But it's important to recognize that it's a compromise in design that inherently reduces your flexibility — such as in this particular case. I don't think there's anything technically preventing Go from supporting non-blocking I/O, though.

Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that? One major difference is that reads can fail with an error, and channel reads can't.

Edit: Also, I didn't mean that select{} on a file descriptor ought to actually return data. It would be more appropriate for it to yield a true/false value to signal that the FD has data, so you'd do "case <-fd: fd.Read(bufSize)".

Agree with everything. Just to reply to this:

> Channels and FDs/sockets are both conceptually multi-valued; what did you mean by that?

I was trying to preempt pedantry, but ended up being the pedant myself :P "You can simulate multi valued responses in a channel by passing a struct."

You can select on the read and an time.After channel which "interrupts" the select for a timeout.

You can't select on a read, though. select{} only works on channels. You can run the read in a separate goroutine, but that doesn't stop the read from happening.

You can't use the language "select", but you can call the select(2) system call.

Indeed, someone already pointed this out: https://news.ycombinator.com/item?id=13332847.

How is your "case <-fd: fd.Read(bufSize)" not a race condition?

Race condition? It's no different from select() followed by recv() in C. But I'm talking about semantics here, not suggesting that this would necessarily be how you'd write it. The main idea was allowing select{} to work on non-channel sources of notifications, such as file descriptors.

> I'm not a Go apologist by any means, I think it has a tremendous amount of flaws.

Hopefully not to derail this too much, but what do you see as the major flaws?

I'm asking out of curiosity - not trying to start a flame war.

There's a problem lurking behind this "simple" model: the goroutine you park on that blocking IO operation can't be collected reliably itself.

I care about collecting goroutines, and would advocate that everyone should care about this. There's been a litany [1] of blogs about this recently, if anyone doesn't consider this self-evident. It's just systematically sane. Letting goroutines spin off into the distance with no supervision is extremely likely to represent either a resource leak, a missing error handle path, or an outright logic bug.

In a dream world, we might have Erlang-style supervisor trees so this can all be done with less boilerplate. It's almost even possible to do this as a library in go! ... except for this blocking IO problem, which throws a wrench in things: any supervisor of a goroutine that may be IO-blocked may itself be indefinitely stuck, and so on up-tree, making the concept unworkable.

---

[1] Some recent articles hammering the point of "collect your goroutines":

- https://dave.cheney.net/2016/12/22/never-start-a-goroutine-w...

- https://rakyll.org/leakingctx/

It's particularly a problem with timeouts and file descriptors.

That's also a great point. I also wish for supervisor trees and killable processes.

This is an attempt to implement supervisor trees in go as a library: https://github.com/polydawn/go-sup (Hej; author.)

It takes the Wait-for-children behavior path by default, and that seems much safer to me. The downside is, as we're describing: if you do get stuck in IO somewhere, you're a pretty unhappy camper. This library makes the choice of (by default; it's a handler func you can override) logging warnings to stderr if a task doesn't exit within 2 seconds of being told to quit... but this still feels like something of a hack; more of a debug measure than a concrete way to improve system stability. Better ideas would be highly welcome!

That looks very promising, actually, thanks. I'm defininitely going to try it out the next time I have a problem involving a graph of workers.

So, soon I will have some stuff to point you to with regards to how Rust is doing it, but we're not _quite_ there, so as a summary...

With Tokio, you write non-blocking IO stuff as "futures", similarly to promises in JS. These then get compiled down to what's effectively a state machine, and they run in an event loop on top of epoll/kqueue. That event loop is single threaded, and fully non-blocking.

I am not a super, super expert on these low-level details, but I believe that this should mean that it is closer to the C code than the Go code.

Interesting solution!

I like futures, though cancelation has historically been a controversial topic there, too (at least in the JavaScript world; JS promises are not cancelable, which is problematic for HTTP requests [1]).

I do like that you guys chose to not make green threads a language-level construct (as Go did with "go" and goroutines), which means you have the option of supporting multiple models, for the programmer to choose according to their particular use case.

[1] https://github.com/whatwg/fetch/issues/27

Cancellation falls out of the model: you drop the future, and it Just Works. (drop being the regular old Rust destructor stuff, which happens when it goes out of scope, or when you call std::mem::drop.)

Yup! The danger there is that the ecosystem may fragment, however, basically everyone is rallying around tokio, including the other people who were working on similar libraries, so in practice, this should be fine.

This sounds similar to what we have in Nim right now. There are multiple layers (from highest to lowest):

* Async await

* The async dispatcher (event loop)

* Selectors module

Each builds on top of the layers below it. The selectors module is an abstraction for the various system `select()` mechanism such as epoll or kqueue. If you wish you can use it directly, and some people have done so successfully[1]. But you must be aware that on Windows it only supports `select()`.

The async dispatcher implements the proactor pattern and also has an implementation of IOCP. It implements the proactor pattern for other OS' via the selectors module. Futures are also implemented here, and indeed all the procedures return a future.

The final layer is the async await transformation. This allows you to write asynchronous code in a synchronous style and works similar to C#'s await. The transformation works by converting any async procedures into an iterator.

This is also single threaded and fully non-blocking and does give you a lot of control. In theory you should be able to work with any of these layers interchangeably (as long as you only care about non-Windows platforms).

1 - http://goran.krampe.se/2014/10/25/nim-socketserver/

Yeah we map the IOCP model to a readiness one, so it's totally platform independent. Wonder if you all could do the same.

I believe we already do the same thing. As I've said, we use the proactor pattern and map it onto the readiness one.

I still think that John Reppy's Concurrent ML design is the best thought out way to do non-blocking IO and event handling. There has been a lot of progress on CML but Reppy's 1992 PhD dissertation remains a very readable introduction: http://people.cs.uchicago.edu/~jhr/papers/1992/phd-thesis.ht...

> there is literally no way to poll a file descriptor

https://godoc.org/golang.org/x/sys

https://godoc.org/golang.org/x/sys/unix#EpollWait

https://golang.org/pkg/syscall/

Of course it is a bit of a pain to write OS specific variants if you are building a completely crossplatform solution, but it is supported out of the box:

https://dave.cheney.net/2013/10/12/how-to-use-conditional-co...

That said, I agree [regarding the] puzzling decision to not to treat 'select' as a general contract for all asynchronous ops as that fits perfectly with the CSP approach.

Indeed, though I was referring to select{}, i.e. Go's first-class language constructs.