carllerche

True... correctness is very important! I made mention of loom for correctness in the intro at least.

The goal for this article was primarily to focus on scheduler design.

I'm hoping to get posts dedicated to loom soon. It's been an indispensable tool.

Relaxed ordering impacts the compiler. There are a number of optimizations it cannot make. I am not sure exactly what those are, but it is measurable.

Ah, all the helpers around the "FSM" aspect of things are already exposed as part of the `futures` crate: https://github.com/rust-lang-nursery/futures-rs/

`unsync_load` reads the memory as if it were a normal field (no atomic operation). The `std` atomic types already provide `get_mut` which gives access to the "raw" field when the caller has a mutable reference to the atomic. The mutable access guarantees there are no concurrent threads that can access the field, so it is safe.

The `unsync_load` is used because, at that point, two things are guaranteed: - The only thread that mutates the field is the `unsync_load` caller - All other threads will only read from that field.

Because of this, we can just do a "plain old" access of the field w/o the atomic overhead.

As for whether or not it should be in `std`, I don't know. It probably could, but it doesn't have to. It's a pretty "advanced" access strategy.

My understanding of that PR is it relates to how the Go compiler does code generation. Rust takes a different approach and is about to ship `async / await` which is a different strategy.

Preemption is out of scope for Tokio as we are focusing on a runtime to power Rust async functions. So, for the foreseeable, Tokio will using "cooperative preemption" via `await` points.

> So the question here is why not to expose generic state machine framework (FSM/HSM) which can deal with those protocols but is also very useful for many other tasks as well?

Short answer is: you can.

Tokio is more like "node.js for Rust" or "the Go runtime for Rust".

The bit about Tokio & futures is how you use Tokio today. Tasks are defined as futures, and Rust futures are FSMs. So, a protocol can be implemented as a FSM completely independently of Tokio / futures and then used from Tokio.

In Tokio's next major release (coming soon), it will work with Rust's async fn feature (which is also about to hit stable).

Hope that clarifies things.

"Node for Rust" is a good way to think about it!

From what I know about Akka, I believe Akka does more than Tokio. For example, Akka manages a supervision tree. I believe Akka is trying to be more of an actor system where as Tokio is just trying to provide an asynchronous runtime for Rust.

Thanks! I agree with your assessment of the Mutex. Improving that is included in the follow up work.

I could have kept working on this for many more months... but my collaborators were starting to get grumpy and telling me to ship already :)

Author here. I'll be watching the comments and can answer any questions!

I do want to make clear that most of the optimizations discussed are implemented in the Go scheduler, which is where I discovered them. I wrote the article to call them out as they were not easy to discover.

From that section.

> There are multiple processors, each running on a separate thread.

It's probably easy to miss, do you have thoughts on where else this can be clarified?

That makes sense to me :) Excited to see that happen!

There is one processor per thread.

Sorry, the article must not have been clear. If you can point out where in the article you got the impression there were multiple processors per thread, I can try to update the article to fix.

> tokio, wasn't very efficient at changing events in the queue (requiring a an extra self signal to get the job done).

Could you expand on that?

How is the topology discovered without tracing? The article mentions that metrics are used. Could you elaborate on how you go from metrics to the topology?

Also, a lot of the work on Tokio [1], Tower [2], and the Rust HTTP/2.0 implementation [3] is being driven by linkerd 2.0 (and sponsored by Buoyant).

[1]: http://github.com/tokio-rs/tokio [2]: http://github.com/tower-rs/tower [3]: https://github.com/carllerche/h2

I would say that you could start using it if:

* You are ready to become an active contributor to the project.

* You are brave!

I had good luck. As mentioned above, finding a good GC is not easy because they do not advertise. It is entirely word of mouth.

I live in the pacific NW where there are a lot of old craftsman style homes. High quality woodwork details is the theme.

I started by calling high end custom furniture shops and asked them if they could do built in cabinetry work. They did not, but they recommended a number of woodworkers that moved on from furniture into contracting (carpentry, custom window sashes, storm windows, etc...).

Then, I asked them what GCs they have worked with in the past and which they would recommend.

A few names came up often, I spoke with them, picked one, and was really happy with the work.

I guess the tl;dr is, good GCs use highly skilled subs, so find the subs then ask which GCs they recommend.

You can get this working with Tokio if you know what you are doing.

However, I personally advise against it as I have found that deferring to the OS for scheduling results in poor thread affinity (your state gets bounced around threads unnecessarily).

You generally get better throughput by either running multiple fully isolated reactors (the seastar approach) or running a single reactor which only flags tasks as ready to be executed, and then use a work stealing thread pool to do the actual execution.

Both of these structures are easy to achieve with Tokio.

Tokio is about being flexible (true today, even more true in the upcoming release). It is more about set of primitives that you can assemble in a way that fits your needs. You can structure the concurrency of your application however you want.

So...

> High I/O systems usually spawn multiple reactors, e.g. one reactor per CPU core.

Depends on what you are calling multiple reactors. If you mean a loop that responds to events and run tasks, then yes. For example, you can plug in [this](http://github.com/carllerche/futures-pool) as the task executor and get a multi threaded, work stealing, scheduler.

Or, maybe you are talking about OS level selectors (epoll), in which case you are going to run up against OS limitations.

Yes you can do this. The actual socket is not pinned to any thread, so you can move it to a thread pool or whatever.

I'm (one of) the author of Tokio, hopefully I can clarify some points.

> Unfortunately, Tokio is notoriously difficult to learn due to its sophisticated abstractions.

IMO, this is largely due to the current state of the docs (which are going to be rewritten as soon as some API changes land).

The docs were written at a point where we were still trying to figure out how to present Tokio, and they ended up focusing on the wrong things.

The Tokio docs currently focus on a very high level concept (`Service`) which is an RPC like abstraction similar to finagle.

The problem is that, Tokio also includes a novel runtime model and future system and the docs don't spend any time explaining this.

The next iteration of Tokio's docs is going to focus entirely at the "tokio-core" level, which is the reactor, runtime model, and TCP streams.

tl;dr, I think the main reason people have trouble learning Tokio is because the current state of the docs are terrible.

> Aren’t abstractions supposed to make things easier to learn?

Tokio's goal is to provide as ergonomic abstractions as possible without adding runtime overhead. Tokio will never be as "easy" as high level runtimes simply because we don't accept the overhead that comes with them.

The abstractions are also structured to help you avoid a lot of errors that tend to be introduced in asynchronous applications. For example, Tokio doesn't add any implicit buffering anywhere. A lot of other async libraries hide difficult details by adding unlimited buffering layers.

One can spawn as many reactors as you would like. The only thing a reactor does is receive events off of epoll (or other system selector) and notify the associated task. The task could be on the reactor thread or across a different thread.

Generally speaking, how to optimize concurrency for a network based application is pretty use case specific.

tl;dr, you can fully take advantage of many core systems w/ Tokio.

The futures underpinning of Tokio actually doesn't have much to do with futures. The task system is really what is special about it (though, currently futures are the main way it is exposed) however, there is a lot of potential other routes that can be explorered. If you want green threads, you could build them on top of the task system and it would interopt with futures.

I'm curious what gives you that impression. What do you think is missing or should be tackled sooner than later? There has been a lot of evolution, however it has all been done in backwards compatible ways so there has not yet been a need to bump the version. It's also spread out across a number of crates.

Rust uses type inference to figure that out. Because `addr` gets passed to `TcpListener::bind` which requires a `SocketAddr`, the type inferencer knows that `addr` should be a SocketAddr and so it all works :)

Hey, I'm the one who received the MOSS award to work on Mio. Funny hearing about this today.

The goal of the Mio work is, in part, to implement a higher level, future based API on top of Mio. The award has been amazingly helpful to allow me to focus on getting it done.

I actually announced Tokio on Wednesday, which is the result of this work:

Announcement: https://medium.com/@carllerche/announcing-tokio-df6bb4ddb34 Code: https://github.com/tokio-rs/tokio

I think the work is headed in a good direction. I introduced a `Service` trait similar to Finagle to make working with services easier. I also provide a new streamlined way to write non-blocking IO services. The blog post describes this in more detail.

I'll be around if there are any questions about Mio & Tokio.

There is a lot in the pipeline! I'll be talking more about it soon.

Replying here since I can't reply to your deeper comment.

A client can never know with certainty that it has successfully acquired the lock because, for example, the process could freeze between reading the current time from hardware and the steady_time function returning.

You can not rely on time, for safety. Only for advice.

References to what specifically?

Martin provided a number in his original blog post, I also elaborated a bit more on my various points in other comments.

In short, either the storage layer provides enough guarantees to ensure safety without redlock OR it is not possible to ensure safety.

I believe that antirez's proposal is the former, but if he has a safe algorithm that does not require a linearizable CAS operation in the storage layer and is safe w/ redlock, I would love to hear it.

I also provided an example of how to use Cassandra which is eventually consistent safely with a locking service that provides monotonic tokens.