there are even images of it on our website!
269 karma · joined December 20, 2017
Perhaps the full example should have been reproduced in the RFD for clarity…
How does that prevent this kind of deadlock? If the owned future has acquired a mutex, and you return that future from the select so that it might be polled again, and the user assigns it to a variable, then the future that has acquired the mutex but has not completed is still not dropped. This is basically the same as polling an `&mut future`, but with more steps.
On the other hand, it also wasn't our coworker who had written the code where we found the bug who was to blame, either. It wasn't a case of sloppy programming; he had done everything correctly and put the pieces together the way you were supposed to. All the pieces worked as they were supposed to, and his code seemed to be using them correctly, but the interaction of these pieces resulted in a deadlock that it would have been very difficult for him to anticipate.
So, our conclusion was, wow, this just kind of sucks. Not an indictment of async Rust as a whole, but an unfortunate emergent behavior arising from an interaction of individually well-designed pieces. Just something you gotta watch out for, I guess. And that's pretty sad to have to admit.
Go, unlike Rust, does not really have a notion of intra-task concurrency; goroutines are the fundamental unit of concurrency and parallelism. So, the Go runtime can reason about dependencies between goroutines quite easily, since goroutines are the things which it is responsible for scheduling. The fact that channels are a language construct, rather than a library construct implemented in the language, is necessary for this too. In (async) Rust, on the other hand, tasks are the fundamental unit of parallelism, but not of concurrency; concurrency emerges from the composition of `Future`s, and a single task is a state machine which may execute any number of futures concurrently (but not in parallel), by polling them until they cannot proceed without waiting and then moving on to poll another future until it cannot proceed without waiting. But critically, this is not what the task scheduler sees; it interacts with these tasks as a single top-level `Future`, and is not able to look inside at the nested futures they are composed of.
This specific failure mode can actually only happen when multiple futures are polled concurrently but not in parallel within a single Tokio task. So, there is actually no way for the Tokio scheduler to have insight into this problem. You could imagine a deadlock detector in the Tokio runtime that operates on the task level, but it actually could never detect this problem, because when these operations execute in parallel, it actually cannot occur. In fact, one of the suggestions for how to avoid this issue is to select over spawned tasks rather than futures within the same task.
Rust async is, however, very useful in single-core embedded systems that don't have an operating system with preemptive multitasking, where one thread of execution is all you ever get. It's nice to have a way to express that you might be doing multiple things concurrently in an event-driven way without having to have an OS to manage preemptive multitasking.
In the case of `select!`, it is a direct consequence of the ability to poll a `&mut` reference to a future in a `select!` arm, where the future is not dropped should another future win the "race" of the select. This is not really a choice Tokio made when designing `select!`, but is instead due to the existence of implementations of `Future` for `&mut T: Future + Unpin`[1] and `Pin<T: Future>`[2] in the standard library.
Tokio's `select!` macro cannot easily stop the user from doing this, and, furthermore, the fact that you can do this is useful --- there are many legitimate reasons you might want to continue polling a future if another branch of the select completes first. It's desirable to be able to express the idea that we want to continually poll drive one asynchronous operation to completion while periodically checking if some other thing has happened and taking action based on that, and then continue driving forward the ongoing operation. That was precisely what the code in which we found the bug was doing, and it is a pretty reasonable thing to want to do; a version of the `select!` macro which disallows that would limit its usefulness. The issue arises specifically from the fact that the `&mut future` has been polled to a state in which it has acquired, but not released, a shared lock or lock-like resource, and then another arm of the `select!` completes first and the body of that branch runs async code that also awaits that shared resource.
If you can think of an API change which Tokio could make that would solve this problem, I'd love to hear it. But, having spent some time trying to think of one myself, I'm not sure how it would be done without limiting the ability to express code that one might reasonably want to be able to write, and without making fundamental changes to the design of Rust async as a whole.
[1] https://doc.rust-lang.org/stable/std/future/trait.Future.htm... [2]: https://doc.rust-lang.org/stable/std/future/trait.Future.htm...
A big part of what we're offering our customers is the promise that there's one vendor who's responsible for everything in the rack. We want to be the responsible party for all the software we ship, whether it's firmware, the host operating system, the hypervisor, and everything else. Arguably, the promise that there's one vendor you can yell at for everything is a more important differentiator for us than any particular technical aspect of our hardware or software.
The motivation, though, was that we wanted to write tests for the function that would run on a developer's machine using `cargo test`, and the Hubris kernel currently only compiles for the targets that we actually run Hubris on (various Cortex-M targets). So, if we want to write tests for that function, we would either need to move it to a crate that doesn't contain Cortex-M-only code, or litter the whole kernel with `#[cfg(not(test))]` attributes so that most of it doesn't compile when building tests. This felt much less unpleasant than conditional compilation. We're hoping that, eventually, other complex-but-not-architecture-specific kernel code will end up in the `kerncore` crate as well so that we can write tests for it, too, so eventually it won't be a crate with only one function...
I do think that there's room for tooling improvement to make writing host-platform tests for `#![no_std]` Rust that gets cross-compiled to another platform, but I don't have any particularly concrete ideas for what that would look like. For now, at least, putting it in a separate crate lets us have our tests --- and those tests let us ensure that some of the function's edge cases are handled correctly, so I do think it was worth it.
if you’re wondering why it’s weird, half-finished, or under-documented, it’s because i wrote it in a couple hours to scratch my own itch, and really didn’t expect to be at the top of hackernews today! if this is something that other people are actually interested in using, i’d be happy to clean it up a bit and add some of the missing stuff…
We're also thinking about factoring out the Tokio Console command-line application's internal data model and client code into its own library (https://github.com/tokio-rs/console/issues/227) to make it easier to build other UIs on top of that.
Side note, it turns out that detecting what color palettes a terminal supports 24-bit colors is surprisingly fraught. There are a couple env variables that may be set...but not every terminal emulator will set them. And then you can use `tput`...but the terminal may not have correct data in the tput database. So that was fun to learn about!
In practice, I personally just use REPLs mostly for quick testing out of a small expression or something...and honestly, I usually just use the Rust playground (https://play.rust-lang.org/) for this. Small examples are compiled fast enough in the playground that it's kind of a REPL-like experience for testing stuff out semi-interactively...but it's not the same as connecting to a running application and running new code inside of that application. That's something that seems very difficult to add to Rust, a compiled, statically-linked language with limited support for hot reloading...
I've also recently started accepting donations on my personal GitHub Sponsors page (https://github.com/sponsors/hawkw) if you're interested in supporting my open-source work in particular.
You can follow the progress of that here: https://github.com/tokio-rs/tracing/pull/1608
I believe it's currently just waiting for a crates.io release of `valuable`!
Regarding the color scheme, glad you like the idea. Because it's a terminal application, the choice of the colors was constrained a bit by the ANSI 256 color palette (https://www.ditig.com/256-colors-cheat-sheet); I wanted it to be obviously a gradient, so I just picked colors that were immediately adjacent to each other in the ANSI palette. It might be better to pick colors that are one step apart from each other, instead, so they're more distinguishable visually...but there's kind of a balancing act between distinguishability and having a clear gradient. We'll keep playing with it!
`tracing` instruments futures by wrapping them with a future combinator that enters a span each time the future is polled; the `#[tracing::instrument]` attribute will do the same thing under the hood when used on an `async fn`. This is kind of analogous to the Go-style context parameter, in that the contexts are stored in structures or on the stack, except that users don't have to manually pass the context around.
The core library provides an option to set the `Subscriber` that collects trace data in a scope; this does use thread-local storage. However, the default dispatcher can also be set globally (like the `log` crate), and the use of thread-locals is feature-flagged so it can be turned off by `no-std` users.
Finally, I have some thoughts on an abstraction for "context-local" storage that allows the user to customize the context that's used to shard the data. This could be used like a user-space version of OS thread-locals when threads are present, but it could also be used by bare metal code for (say) having a context for each CPU core. This would allow subscriber implementations to track a span per thread by default, but let embedded or kernel-mode users override this without having to reimplement the rest of the subscriber logic. This is still in the early stages though.
Hope that all makes sense; I'm happy to answer any further questions!