Node.js Flow – Callback Hell vs. Async vs. Highland Streams | Hacker News Reader

Node.js Flow – Callback Hell vs. Async vs. Highland Streams

blog.vullum.io

91 pointsddispaltro63 comments

Unfortunately all those

    .flatMap(_.wrapCallback(one))
    .flatMap(_.wrapCallback(two))
    ...

    .then(Promise.promisifyAll(one))
    .then(Promise.promisifyAll(two))
    ...

are noise making the signal difficult to read. It should come down to nothing more than

    one | two

plus the error handling clause. That's not JavaScript but you got the idea. Syntax is important. I still have bad dreams about arrays() hell in an app written by a customer with CakePHP 1 (a Rails clone with an approach that didn't suit PHP well). I'll skip over the $this-> mess.

I'm not much into asynchronous languages. Is there any examples around with a built in and slim syntax for concatenating callbacks? Thanks.

In Haskell it would be

    do one
       two

because it's pretty well understood there that syntax can be a big differentiator.

What's more is that this syntax adopts for the binding aspect of flatMap. So if two is dependent on one...

    do output <- one
       two output

which is a shortcut for

    one >>= two

which might be

    Promise.promisifyAll(one)
    .then(function (output) {
       return Promise.promisifyAll(two(output)) 
    })

in Javascript

Haskell has the advantage that what you've proposed is an exact conversion of the "flatMap" code, meaning that anything else that you can do with flatMap can be used with do notation, which is a lot of things beyond just tying two bits of IO code together.

But in this specific case of just tying two bit of execution together, I'd observe that Erlang is

    One(),
    Two()

and Go would be

    One()
    Two()

and that's all regardless of whether One and Two have 0 accesses to IO or do IO dozens of times, and One and Two may also freely call other functions that do arbitrary IO with no further ceremony. The "solution" Erlang and Go offer to callback hell is simply not to have it, and pretty much no matter how you slice it, not having the problem in the first place beats any solution the JS community may or may not settle on.

That's completely fair and accurate. The challenge is handling the genuine concurrency model then, but both Go and Erlang have good tools for doing just that.

So it kind of comes down to whether you want to shave off ceremony for concurrency or for all monadic/sequencing operations at once. Honestly, that's a tradeoff.

> I'm not much into asynchronous languages. Is there any examples around with a built in and slim syntax for concatenating callbacks? Thanks.

Yeah it's called any language that offers good abstractions for concurrency. Elixir/Erlang, Haskell, Clojure and Go are a few good examples.

I played a little with Elixir (and Phoenix) months ago and I know the |> operator, which is not exactly what I was looking for in my reply. I'll look again at Elixir and at how it handles callbacks.

There aren't any callbacks in Elixir. The run time takes care of scheduling for you. The code you write looks synchronous but is executing asynchronously.

Not only does this make your code easier to read, it also means that the entire ecosystem is available to you rather than a partitioned between sync and async libraries.

|> is (not, see below) mere function composition

    function compose(f, g) {
      return function inner(x) {
        return f(g(x));
      }
    }

this is not the right kind of composition for something like nested callbacks. For that you need slightly more.

Edit:

It's actually the "apply" operator which looks like

    function apply(f) {
      return function applied(x) {
        return f(x);
      }
    }

Apply and compose work together nicely. For instance, in Haskell apply is ($) and compose is (.), so you can build a compositional pipeline and apply it like

    f . g . h $ x

or like

    f $ g $ h $ x

since association goes this way

    f $ (g $ (h $ x))

Deleted comment

Ah, my bad, I was distracted writing. It's Haskell's ($), not the (.).

No worries. I deleted my post before your reply came through because I didn't feel my example was good and I'm struggling to do better this early in the morning :D

In a lot of cases compose and |> will be similar due to operations that are commutative, but they are semantically different.

Rust is also worth a mention as one to watch.

Recently Rust removed libgreen from their standard library and went back to native threading so that users can choose and build their own abstractions. So as a language by itself Rust doesn't have a ton of concurrency abstractions but the community appears to be working hard creating concurrency abstractions and I'm sure it's only a matter of time.

To expand on this a bit, while they went back to native threading, they didn't ditch any of the type system and library work designed to make message passing convenient and memory safe, which is (arguably) more fundamentally important for concurrency than M:N threading.

Great point. I just meant in the context of the languages I cited Rust doesn't offer similar abstractions.

Yeah not disagreeing with you at all. It doesn't work the same way "concurrency-focused languages" generally work right now, but it does share the message-passing philosophy with them. Interestingly, I heard a rumor that Go is considering moving to 1:1 threading on Linux, which would align it with Rust's current approach, just without the memory protection capabilities of the type system (though I have no idea whether or not this is true).

C# has the 'await' keyword, which is pretty clean.

If you're dealing with vanilla node.js streams, you can just do:

   one.pipe(two)

You can also use a transpiler such as 6to5[0] which supports ES7 async/await.

[0]: http://6to5.org/

...and you can also have something like that using Bluebird.coroutine() (https://github.com/petkaantonov/bluebird/blob/master/API.md#...) or co (https://github.com/tj/co, see also https://github.com/koajs/koa)

6to5 actually supports compiling async/await to bluebird coroutines

Whaaa... they're putting experimental ES7 features in a transpiler called 6to5? Or am I missing something?

Its getting a new name: https://github.com/6to5/6to5/issues/596

The project scope has expanded: https://github.com/6to5/6to5/issues/568

Definitely my preferred approach. Note that you need to run 6to5 with the -e flag to try this.

I think you mean the --experimental flag. -e is for evaluate.

Where does it say that it supports async/await?

http://6to5.org/docs/usage/transformers/#async-to-generator

Combining Promises+ES6 Generators seems the next way to go. It is already used successfully in many Firefox tests and the syntax just feels great. For the Node.js ecosystem, you should take a look at IO.js [1], Koa [2] and co [3] that are able to handle nice async syntaxes.

[1]: https://iojs.org/ [2]: http://koajs.com/ [3]: https://github.com/tj/co

I think RxJs[1] can help here as well.

[1]: https://github.com/Reactive-Extensions/RxJS

Interesting comparison. I have never used Node for anything bigger than small personal stuff, because non-blocking code was never a requirement. I still dont see the value in it in scenarios like in the blog post, as all he is doing is a bunch of functions and once all of them are done, send something back to the client.

How does the client or the dev benefit from the non-blocking approach here? At the end of the day he is describing a sequence of actions the depend on each other. Why not block then?

I really dont want to sound arrogant, but I am just curious what the benefits are that I dont know of.

Originally, for performance reasons. Async code lets you handle multiple concurrent connections without having to rely on multiple threads or processes (which are more resource hungry)[0].

Nowadays, there's also the fact that if you want to use Javascript on the backend, Node.js is the de facto standard. You don't necessarily have to do async I/O but you'd be fighting against the whole ecosystem by not doing so.

[0] http://en.wikipedia.org/wiki/Preemption_(computing)#Preempti...

I agree that is is the standard way of doing things. I was just curious why the trend moved towards this approach. Not a long time ago Concurrency based on a lot of threads had an upswing.

http://www.mailinator.com/tymaPaulMultithreaded.pdf

Because node is single threaded, so blocking for this user will block all requests for all users.

That explains it, thanks. To me this makes the focus on non-blocking code in PlayFramework2.0 a bit more strange to me, since the JVM isnt single threaded.

it's because native threads are expensive

They are again? In this talk they mentioned them to be cheap in 2008

    .then(Promise.promisifyAll(process1))
    .then(Promise.promisifyAll(process2))
    .then(Promise.promisifyAll(process3))

Shouldn't it be just Promise.promisify[1]. Not that it matters, promisifyAll does the same thing, but also looks for child functions to promisify.

[1] https://github.com/petkaantonov/bluebird/blob/master/API.md#...

Blog owner here: Yes, you're correct, will fix it asap :-)

fixed :-)

I don't like that highland uses the "_" variable in their examples. The same variable name is already used by lo-dash and underscore.

using generators is much more elegant.

And fibers even more so, in my opinion.

I hate how they can introduce collateral damage to other flows. They’re many times more powerful than generator+yield/async+await, but I love how yield/await is explicitly telling where an async behaviour will be flattened into the routine.

That's a bit of a made up problem. Having used fibers for a few years now for some heavy lifting, I've never had issues with not knowing where async behavior occurs. No different from goroutines really.

I didn't know about fibers. I'll take a look later today.

Noticeably missing from that list is regular Streams2, with possibly Through2 / From2 to make it easier to work with.

There has surfaced quite a few additional approaches now, and I'll try to incorporate all of them into the post.

I started using node.js 1 year ago for a personal project and I must say that the callback hell is a real issue to be solved.

Personally I think the highland approach is too verbose, while the bluebird promises is the one I prefer for both readability and functionality.

Does anyone has used them in big production systems? Are you happy with them?

I use bluebird in all of my production systems, not only because I believe it creates the most readable code, but also because it can do things that callback flow libraries and async simply cannot do.

It's fast to boot and is great for error handling. I highly recommend it.

I like the idea of highland, but I think as a practical matter bluebird promises are the best solution for production.

I've used them a fair bit in production, and I'd say yes, I'm quite happy with them.

Edit: I've used async too, but I feel that async does not scale well in terms of complexity. When you start needing solid error handling while dealing with multiple async processes with complex mixtures of parallel and sequential...bluebird works, async just ends up as a mess. In my experience, anyhow. Either lib works great for simple uses cases though. :)

Thanks for the reply, this was just what I wanted to understand

We use bluebird. With BLUEBIRD_DEBUG=1 we get amazing long stack traces that span across events, and they got even better (cleaner, more accurate) post bluebird 2.8.

If willing to take a performance hit (they will remain faster than caolan's async or native promises though), they're also usable in production.

sounds like a really good feature. I think I'll try bluebird on my personal project to rewrite part of the code. thanks

I've been using async for quite long time now and it's absolutely sufficient - never had any problems, it has all the control flow types you'll need and in my opinion it's very readable. It's also the most mature amongst them, so yes, it's definitiely ready for big production systems ;)

One approach is to use an underlying scheduler so that even while (1) { ... } kinds of things don't lock everything up.

This is the approach that Erlang (and therefore Elixir) takes.

Highland, promises, async.js are all hacks. Best to use a language construct like await/yield, as in ES6, tamejs, and Iced CoffeeScript. These days noone should experience callback hell, it is a well-solved problem.

Not saying you are wrong (I don't know enough to disagree) but if asynchronous program is solved in node/javascript, why is there the need for all these post on callbacks, streams, promises, fibres etc?

Can you expand more on your comment and/or point to some references?

"Callback hell" is very effective linkbait for a title, there is not a need for the post. Promises are an effective implementation of control flow for ES5 JS, but don't really improve program structure. They are most useful "under-the-hood", as TameJS/Iced-CS creates when compiling.

The overarching solution is to be "flat" by using small, composable functions. Lazy.js and Highland.js force the user to do this but require learning a specific syntax. Streams make this technique easier to reason about (easier say, than Transducers or Generators), but the goal is the same: operate on one object at a time with one purpose, and return a result in a standard manner so it's easy to consume with the next function. This is good practice, but leaning on one of these libraries is an excuse to force the practice.

Chained callbacks is JS looking from the inside out; looking from the outside in, we see this practice is solved by many other languages [1] -- not just serial calls but parallel also! C#, Scala, Python, Java, Dart, Spiderjs and more have implementations of Futures [2].

Fibers implement Futures but require a native module for V8 -- since they are not portable outside Node they don't have great adoption and aren't used in almost any node.js modules: only ~100 out of the endless ecosystem [3].

Fibers, Async.js, and Highland also aren't syntactically an improvement over Promises -- the point of programming, which devs will get tunnel vision and forget -- is to leverage the language to do more work for less keystrokes/manhours [4]. The compiler should create control flow for my code, not the other way around. There should be no reason to learn a complex API just to get serial/parallel calls, when a simple keyword addition to the language will suffice.

Not to mention, await/defer is so readable that new devs can start working with async code right away, without needing to research. This is not the case with promises/fibers/asyncjs/highland or whatever flavor-of-the-month a shop has picked for control flow.

[1] https://github.com/KjellSchubert/promise-future-task

[2] http://en.wikipedia.org/wiki/Futures_and_promises#List_of_im...

[3] https://www.npmjs.com/package/fibers

[4] http://blog.teamleadnet.com/2013/08/async-await-in-net-is-no...

No mention about NoFlo, the Node.js Flow-based programming library, designed to tackle exactly this problem? [1]

[1] http://noflojs.org

Difficult to evaluate it when I couldn't see a single line of code on that website. It looks to me that it's targeting quite a different problem.

vs. using a real language?

Webbies seriously have Stockholm syndrome with this language.

Reply on news.ycombinator.com