Do this streams have any notion of back pressure? I can't find info on that.
For real world distributed systems back pressure is very important. Otherwise you could DoS yourself by some cascading overload.
For real world distributed systems back pressure is very important. Otherwise you could DoS yourself by some cascading overload.
Of course, depending on the rest of the system you could still have a bottleneck somewhere. For example: if you pull jobs from a queue in (say) Redis but don't have enough capacity to process jobs faster than they are enqueued, the queue will eventually fill up.
So it's pull based and not push based like most other streams lib.
Does maybe someone know how this compares to FS2 or Iteratees than? (Both are also pull based streaming solutions).
https://en.wikipedia.org/wiki/Iteratee
Looks quite similar to me. Is the Scala FS2 lib maybe even a clones of the Haskell solution? Or are they different in important aspects?
For parallel processing the backpressure becomes a bit more complicated, in particular if you end with a sequential consumer. If you're just doing your parallel tasks when a sequential consumer demands it, you'll find yourself still doing one task at a time. The straightforward solution is to work ahead a bit, spawning just enough parallel work to fill a small buffer that the consumer can read from. You get backpressure this way, but may have done a little bit too much if the consumer doesn't want the rest of the data. We'll show this in the third blog post of the series.
Thanks for the pointer! Will have a look.