Parallel-Stream | Hacker News Reader

Parallel-Stream

blog.yoshuawuyts.com

This is pretty cool :) Not sure if author is here, but I had some questions:

- How do exceptions work in parallel streaming? How might async complicate things?

- What kind of logging / instrumentation is available? Can you use trace?

- Is there a result set generated at the end like Trio nurseries, or are results meant to be handled by another process?

I know nothing about Rust or low-level systems programming, but I should go and learn it (and a bunch of other stuff too).

author here on HN is yoshuaw (great guy btw!), someone should ping him on Twitter that his project hit HN frontpage and I'm sure he'll find you :) https://twitter.com/yoshuawuyts

Thanks for the kind words; I just got a ping on Twitter and found my way here, almost certainly because of your comment. Thank you (:

- Rust doesn't really have exceptions is the short answer. Errors are represented by the Result type.

- You can trace inside of async code, there are libraries like tracing and tracing-futures.

- Not sure how to answer your last question. From my understanding of this library is uses the async-std runtime, which comes with a threadpool capable of executing many tasks over a set of processes. Each of the async blocks you write inside a combinator here (.map, .for_each, etc) could hypothetically run on any thread, the idea is each task is running in parallel. That's what I'm gathering anyway

Hi, author here. Glad you've enjoyed the post! To answer your questions:

As leshow pointed out, Rust uses the Result type to propagate errors. The way we'll integrate this in parallel-stream is by adding `try_for_each`, `try_fold`, and `try_map` methods. These methods will short-circuit when an error is returned, and have counterparts in `std` and `async-std` already. One tricky part is to get cancellation right, but we already have a design for that, it's just a matter of implementing it now.

`async-std` comes with support for the `log` crate built-in, which provides information when a new `Task` (lightweight thread) is spawned. `parallel-stream` doesn't perform any logs on top of that, but it probably should! I'll open an issue.

Result sets can be generated through the `collect` method which calls the `FromParallelStream` trait. This can be implemented for various types, but common examples include: `Vec<T>`(generic array), `HashMap`, and `Result<Vec<T>>`. If you're interested in how this exactly works this post by jcoglan goes into incredible detail: https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust

I hope this answers your questions!

Given only the thread would die, what about panics?

For example, `.parallel_stream().map(n => 1 / n)`

What happens to the `n=0` case? Does parallel_stream propagate the panic, or replace the value with a Result::Err?

It relies on async-std's behavior, which will propagate the panic.

This looks cool. Something I really never get with async is the executor:

- if I use this do I have do use tokio or async-std ? none ? is the code assuming an executor ?

- e.g can I use this library with, let's say, hyper ? tide ?

This library assumes `async-std`; the way it parallelizes work is by spawning tasks on an executor, so by design it needs to have access to one. Without that we couldn't provide the API we do.

I haven't tested it with `hyper`, but it should soon be compatible with `tide`(assuming `async-h1/async-std` is used as the backend).

Reply on news.ycombinator.com