Announcing the tokio-io Crate | Hacker News Reader

Announcing the tokio-io Crate

tokio.rs

136 pointsmiqkt32 comments

Open articleView on HN

I would really like a better information around the difference between Tokio & Futures:

* What is Tokio & why was there a need for its own terminology? (Reactors, Core, Proto etc..)

* What are the differences between Futures and Tokio? When would I use one? When would I use the other?

* Can I use futures without tokio?

* For all of the structs in Futures, Streams, Sinks, etc.. where are the examples?

Even what IS tokio for us casual observers would be nice to know! Futures I can infer, but tokio? I have no clue and the article doesn't help.

https://tokio.rs/blog/tokio-0-1/

Might help illuminate some things?

This post was intended for Tokio users, who would know what it is already. The author even said "this probably isn't relevant to HN" to me.

The main website shows this off, and it's the project's blog. I don't think every single post on a development blog needs to re-hash the whole project.

Whenever I see a blog post like this, if I don't know what the project is, I click the "Home" link to see if the homepage offers a more detailed description. I'd hope Tokio's is (relatively) clear in the detail it provides, is it not?

Tokio is "do that network transfer or disk read without freezing the program".

Except that takes a specific tokio package for each operation rather than being a kitchen sink of everything in one crate.

Oh so tokio is just the generic interface for any "do thing x without freezing", where the actual x comes from whatever implementation you include.

That's my understanding. It's a generic framework/interface for doing any kind of IO task without freezing the program.

As far as I understand, it's like this:

* Futures is a crate that provides a common abstraction for delimited continuations (JavaScript Promises).

* Tokio-Core provides an event loop abstraction (can't remember for sure but I think it uses Mio under the hood to abstract out epoll/kqueue/IOCP). Additionally, it provides the "core language" of Tokio, which are Futures (basically a Result that returns asynchronously), Streams (an Iterator that yields values asynchronously), and Sinks (a place to send data asynchronously).

* Tokio-Proto is an abstraction over Tokio-Core for composing various protocols together. This is done by defining protocol "codecs", which are used to define Streams and Sinks that send some unit of data along the protocol, and a Proto object that ties the codecs together. Then, to write servers with that protocol, you implement the Service trait, which can work at the higher-level protocol units (requests, responses, etc). If you're curious how this looks in an HTTP server for example, check out this (shameless plug) reverse proxy I wrote with Tokio: https://github.com/moosingin3space/hyproxy

* You would use Tokio for any asynchronous I/O task. You would use Futures whenever you want some sort of delimited continuation.

* You can use Futures without Tokio.

* The examples are mostly in the documentation here: https://tokio.rs/docs/getting-started/tokio/

If I'm wrong about any of this, please, someone, let me know!

> * Futures is a crate that provides a common abstraction for delimited continuations

I'm familiar with Java Futures, but can't tell if it's the same concept as what you're talking about. What exactly is a "delimited continuation"?

The best analogy I can give is to JavaScript promises, where you can do something once the future is resolved. In Java, this would be like the CompletableFuture class's thenApply function -- where you can write code that picks up once the future has been resolved.

A "delimited continuation" is that basic idea of having your code pick up once a future finishes executing.

They are roughly comparable, yes. Here's the explanation of futures specifically https://tokio.rs/docs/getting-started/futures/

Something like a -> (b -> r) -> r/the Cont monad?

E: I meant to reply to a a sibling.

You've got one good reply already, let me give it a shot too:

Tokio is an asynchronous IO stack for Rust. Like many frameworks/large libraries, you have to call things something. It's largely inspired by Finagle, and so shares some of its terminology, but not all of it, as Rust and Scala/Java are very different.

You can use futures without tokio. Futures are the lowest level of the stack: they give you a way to create a value that represents something "in the future". You make a future, it processes stuff in the background, and then you call a function to say "wait until the value is ready." You can do stuff in the meantime. It's not tied to IO or anything else, any kind of computation you want. It is mostly a very thin interface. I have a hobby OS project, and I'm considering making it "everything is a future".

mio is a thin abstraction over epoll/kqueue, and a slightly larger one over IOCP. That is, low-level asynchronous IO primitives.

tokio-core is futures + mio. That is, it takes the futures pattern and applies it to IO. It does this by adding an event loop, a "reactor core" (hence Core) that drives the futures along.

On top of that is tokio-proto. This lets you build protocols nicely on top of tokio-core.

This new package extracts out the main interface from tokio-core, so that if you wanted to, you could replace it with something else. Previously, you got these interfaces only from tokio-core, and so you had to depend on it to use them, which means you had to use only tokio-core. The idea with this package is that it gives you more composability; the interfaces are in here, so those working from above can use them, and then when you're building the stack, you can use tokio-core, or some other backend.

Finally, many people who use tokio won't actually use this directly: it's something other library authors build on top of. For example, hyper uses the tokio stack to do async HTTP requests. So most people don't even need to know or think about any of this. For example, I'm working on a little web framework on top of all of this, here's what it looks like to use: https://github.com/rust-lang-nursery/thanks/blob/master/src/... (this is _extremely_ small and a work in progress, don't expect Rails.) No need to look at or think any of that stuff.

As for examples, https://tokio.rs/docs/getting-started/tokio/ goes through the entire stack, bit by bit, building it up. It also links to a variety of other examples.

Tokio-Proto also makes it easy to compose a protocol stack. In here https://github.com/moosingin3space/hyproxy/blob/master/src/t..., which is a TLS client implementation for my Tokio-based reverse proxy, the Service implementation is incredibly easy; simply compose the Hyper `HttpConnector` inside the native-tls/tokio-tls `TlsConnector` in a `Service::call` implementation.

Yeah, this is totally true. It's the part I'm least personally familiar with.

That was a good reply! Do you know how much memory this stack consumes? If I had a microcontroller with 150k of RAM, could I use this?

Off the top of my head, no, but in general, it doesn't allocate very much.

>why was there a need for its own terminology?

https://en.wikipedia.org/wiki/Reactor_pattern

The Reactor pattern has been around for more than twenty years. It wasn't something thought up by Tokio.

Has anyone tried using this to implement a game or any complex GUI really in functional reactive style yet?

I've been playing with ScalaRx a bit (not to be confused with RxScala...) and it's a really clean feeling abstraction for GUI, which I normally hate.

It'd be interesting to see similar ideas applied in Rust.

To someone who knows more about Tokio than me: is this even possible with Tokio's current design? Or is Tokio too I/O centered, such that its core abstractions wouldn't be useful in something like this?

Tokio as I understand it is focused on IO; you'd use some other futures-based thing to do this.

Writing a game is quite possible, but most of Tokio's higher-level abstractions are focused on request-response protocols. I've been working on some tooling (https://github.com/sirpent-team/comms) for non-request-response but it's been slow going.

The sample code for those projects are often really cool, but as soon as you get to real world case, you start having problems if you want to be 100% pure.

Imho, for mobile ( ios / android), the problem is that the base sdk aren't thought for reactive style (push vs pull). So all Rx frameworks try to compensate for this mismatch by adding an adaptation layer, and you end up with a really unconvenient code.

Hmm. I have a Framed<TcpStream,MyCodec>. I used to call into_inner() on this to get the stream when it came time to upgrading to a TlsStream<TcpStream>, but that function has gone with this upgrade. I don't see any equivalent function to do this either anymore. Any ideas?

Seems that the removal of into_inner was a mistake. There's a commit from 9 hours ago adding it back:

https://github.com/tokio-rs/tokio-io/pull/29/files

Bytes seems like too many things in order one. It should use http://burntsushi.net/rustdoc/byteorder/

Looks like it does? https://github.com/carllerche/bytes/blob/master/Cargo.toml#L...

Oh :blush:. In that case, odd to reexport a bunch rather than just link API ind doc.

So Codec and UdpCodec from tokio-core are not related ?

Yeah, they seem to be independent traits.

Finally! I was waiting for bytes crate. Really need it for vectored I/O in my project. Super happy

Reply on news.ycombinator.com