Managing Node.js Callback Hell with Promises, Generators and Other Approaches | Hacker News Reader

Managing Node.js Callback Hell with Promises, Generators and Other Approaches

strongloop.com

92 pointsjguerrero47 comments

Disappointing that most promise articles/tutorials reference Q as the preferred promise library, rather than bluebird - https://github.com/petkaantonov/bluebird

Bluebird is not only so fast that it's almost on par with callbacks[1], but also innovates on the ability to filter on catching different error types[2], support for generators[3], context binding[4] among other things.

[1]: https://github.com/petkaantonov/bluebird/blob/master/benchma...

[2]: https://github.com/petkaantonov/bluebird/blob/master/API.md#...

[3]: https://github.com/petkaantonov/bluebird/blob/master/API.md#...

[4]: https://github.com/petkaantonov/bluebird/blob/master/API.md#...

That library looks pretty cool and is fairly popular , i will give it a try.

Bluebird is definitely the way to go, we recently introduced it to an existing 100K LoC code base and we couldn't be happier.

The stack traces and error management is amazing and everything is very fast.

yes! I recently switched from Q to bluebird and it's been great, and really is exceptionally fast. Also the source code is a work of art - extremely highly optimised.

We just started using Q for our all--aaaargh, motherfucker.

Great to see progress being made, I guess. :(

The important thing is you're using promises. The differences between Q and Bluebird are minimal, compared to promises vs raw callbacks.

I just found this after some cursory Googling and haven't tried it at all, but it appears to be a compatibility layer for Q on top of Bluebird (written by the author of Bluebird): https://gist.github.com/petkaantonov/8363789

Yeah, for the work we're doing, promises are awesome. I've been doing node work recently and really it's the only thing I've seen that makes sense to me for managing this.

It's interesting--I'd considered myself pretty OO in my style, pass a message, hope for the best, but the promises stuff is kind of taking that to its natural conclusion.

I would not say this[1] is a minimal difference :P

[1]: https://github.com/petkaantonov/bluebird/blob/master/benchma...

I am pretty certain he means in terms of code delta.

Looking at it, in general, ignoring that I know about Node internals, you know what is the simplest way to solve this "problem"?

It is something like this:

---

    callback(find_largest(get_stats(read_files(dir)))

---

1 line.

If an error happens, an exception should be raised.

The fact that there are multiple such requests happening or that some of those operations have to wait for a select or epoll loop to return, should be handled by an underlying framework. The fact that even with promises you have to write 27 lines of code to do that, should be making you worried.

Please stop spreading FUD.

With promises (Bluebird) this sounds like very little code in practice, assuming promisifyAll has been called the FS module:

- One line to read all files in a directory

- Map them with .map to a .stat call on them, returning array of promises.

- Reduce them to the largest element.

Something like:

    var fs = Promise.promisifyAll(require("fs"));
    var largest =  fs.readdirAsync("/your/file/path").map(function(file){
         return fs.statAsync(file);
    }).reduce(function(x,y){
        return (x.size > y.size) ? x : y;
    });

With arrow syntax:

    var largest = fs.readdirAsync("/your/file/path").
                  map((file) => fs.statAsync(file)).
                  reduce((x,y) => (x.size > y.size) ? x : y);

Alternatively, with Generators using Bluebird promises

     var largest = Promise.coroutine(function *largest(dirname){
        var files = yield fs.readdirAsync(dirname);
        var sizes = yield files.map(name => fs.statAsync(name));
        return files.reduce((x,y) => x.size > y.size ? x : y);
    });

If one error occurs, the promise rejects. VERY far from 27 lines of code. Not to mention you can of course compose these in one line if you extract them to methods like in your code:

    var largest = fs.readdirAsync("/your/file/path").map(fs.statAsync).reduce(largest);

> Please stop spreading FUD.

Was that addressed to strongloop.com? They are the ones that posted an example with promises using 27+/-whitespace lines of code and the initial problem algorithm only had 4 lines.

Sounds like you should write a blog post and not them about using promises.

I don't use Node.js on daily basis and am not familiar with promises so just commented as an outsider based on the article.

Your last one line looks fantastic and makes it clear what is going on. Any reason you can think of why that wasn't the "obvious" solution to the original posters of the article?

rtdsc: > Was that addressed to strongloop.com? They are the ones that posted an example with promises using 27+/-whitespace lines of code and the initial problem algorithm only had 4 lines.

Oh wow, this is my fault, I didn't read what they did there. I have __no idea__ why it took them 27 lines to implement this. It's rather trivial with promises.

- They have a .then followed by a .map there causing nesting for no reason whatsoever.

- They also get the file name, but handle this very poorly (with an array), where they could just do it as a part of the .map (the .all and .then are completely redundant) there

- They assume the possibility of directories, I didn't. That .filter is justified but that's the only thing one might have to add here.

- Generally, it's poor promise code.

I guess that promises are just not their tools of choice and they just used them _for that post_.

> If an error happens, an exception should be raised.

For better or worse, Node's best practice seems to be to propagate errors via the err parameter in each callback. E.g. `function myCallback(err, data)` where data represents the result of a successful computation.

One could reasonably debate whether this was the best possible design. Some people like exceptions, for one thing. For another, JS's dynamic typing, combined with multiple layers of error propagation, could lead to mishandling of the errors--you expect your callback to be passed error A, but it actually gets error B from elsewhere in the callback chain.

But this pattern is ingrained by now. If people started deviating from the pattern and using exceptions, you'd end up with an even worse monstrosity. Every bit of calling code would have to handle both styles of errors.

It's not just a best practice - exceptions raised from async code callback simply do not work right (they'll never make it back to the caller). In practice an exception raised from a callback will take a very short trip the wrong way through the stack (whatever happened to invoke your callback is _not_ the original caller, otherwise you wouldn't need callbacks), and end up as a toplevel uncaught exception, killing your entire process in nodejs.

Good point. I had previously thought the callback in some way "remembered" who called it, and could bubble an exception back up to the caller's try/catch block. But you've set me straight. Further proving the point that you can't just throw exceptions whenever you want.

Don't promises address this exception problem because the top level code is calling the then () function, which can then throw an exception to the appropriate caller?

Yes, I believe so. I was just addressing a the original suggestion, which was to avoid using promises and use vanilla callbacks instead. In that architecture, exceptions don't seem to be workable.

Actually, you can throw exceptions from within .then just fine and they will be always bubbled. You just cant go

  function g() {
    return somePromise.then(function() { 
      setTimeout(function() {
        throw new Error();
      }, 500);
    });
  }
  g(); // crash

But at that point, you're not using promises anymore so all bets are off.

However if you use a promise-based timeout too:

  function f() {
    return somePromise.then(function() {
      return Q.delay(1000).then(function() {
        throw new Error("Oops");
      });
    });
  }
  f().catch(function(e) {
    console.log(e.message); // logs Ooops.
  });

No problem bubbling that.

Promises normalize all exceptions in promise code to bubble up the chain - as such, they do make it back to the caller.

(However, they do leave async exceptions from callbacks alone - only domains can handle those)

It seems to me that Node.js is just manually writing in continuation-passing style, which is PAINFUL. Why not simply use a language which supports continuations natively? Heck, writing a Scheme->JavaScript compiler really shouldn't be that difficult.

You may find StratifiedJS pleasing: http://onilabs.com/stratifiedjs

You wouldn't even need that one callback, just:

return find_largest(get_stats(read_files(dir)));

Exceptions are propagated as you would expect from synchronous code, despite being async under the hood.

The "promises" way of writing that concisely would be:

    read_files(dir).then(get_stats).then(find_largest).then(callback, errback)

Any exceptions in those functions would propagated to the "errback". Not perfect but much better than the examples in the article would suggest.

The generator example has no error handling (at least not from what I can see).

Once the "callback hell" example was re-written with named functions, it is not much different than the generator example and depending on how you are testing, easier to test because it separates out IO from logic (Which you could still do in the generator example).

One thing callback style code has begun to make me think about is the nesting/IO complexity of the functions a I write. When you nest, the IO/event boundary is visible in the code (by the indentation). As you continue to nest, you can start to get a clear idea of the amount of IO being done and where a particular function may be doing too much. Anyhow, most of this is preference anyway :)

A simple try/catch around the whole thing would take care of error handling for the generator example.

only until you have a syntax error :)

As you read all these many lines of code to simply find the largest file in a directory, recall that in any language with friendlier blocking/non-blocking semantics, it could be as simple as (e.g. in Ruby)

    Dir.glob('*').select{|f| File.file?(f) }.max_by{|f| File.size(f) }

Plus, the error semantics of this code are sane: if something goes wrong I get an exception I can rescue. If there were no files in the directory, I get nil.

Statting a bunch of files is rarely so costly that parallelism is warranted, unless it is impressed upon you as in Node. The cost in readability and dreaming up generators and promises to solve such a mundane problem is likely much worse. Before you charge headlong into building your next webapp with Node, consider this article food for thought.

It's definitely a trade-off. What Node gives you is async when you need it at the cost of giving you async even when you don't.

The pain and constraints introduced by this forced use of async sure has spawned a lot of wheels of all sizes, colours, and compositions.

There's always `fs.readdirSync` etc. node isn't forcing async on you, especially when it comes to file handling.

Ruby is the wrong comparison. Write that same code in Go, suitably translated, and it would be fully "async".

I hit Go just because it's the most Algol-ish of the sane languages. There's several other options where you just write the code and the compiler and runtime do the async for you.

It was a short, simple example to illustrate how yield will improve Node. In real life, you'd probably just do something like:

    fs.readdirSync(dir).map(function (f) {
      return {
        name: f,
        stat: fs.statSync(path.join(dir, f))
      }
    }).sort(function (a, b) {
      return b.stat.size - a.stat.size;
    });

Not going to win any code golf tournaments, but that's JavaScript for you. It was never designed to do this sort of thing. Its strengths lie elsewhere.

In real life you wouldn't use sync methods, they would destroy your server's performance by blocking the event loop.

With arrow function syntax, extra `maxBy` and `filter` methods on promises for arrays, and nicer file/dir libraries:

  Dir.glob('*').filter(f => File.isFile(f)).maxBy(f => File.size(f));

Asynchronicity isn't the issue there, the issue is that most JS / node libraries are minimal, not optimized for ergonomic (and of course the verbose lambda syntax)

I remember doing a deep dive into node and evented JS a couple years ago and I realized that I couldn't recommend it to my team because the code would become such a huge mess on an large system that whatever performance you gain, you lose in maintainability in a large team environment.

The best I could find at the time is IcedCoffeescript, which sort of replaces the whole existing coffee script tooling, so now you are dealing with custom tooling to install for each team member as well.

I find it sad that years later it looks like what IcedCoffeescript did is still the cleanest looking solution to the problem. It feels like the evented model is maybe great for smaller things, but horrible for building and reasoning about in larger systems on larger teams.

I'd like to mention my favorite tool to deal with Callback Hell: Streamline, a JavaScript (and CoffeeScript) language extension that can compile to either 1) callback spaghetti 2) Node-fibers or 3) generator code. It basically writes the mess for you. It lets you focus on the higher order concepts and provides configurable parallel array operations as well as many other convenient features. The callback "error parameter" is thrown as an exception to be caught with a basic try/catch. I highly recommend that any Node and JS enthusiast give it a try.

https://github.com/sage/streamlinejs

There is another nice feature of ES6 that helps a lot with callbacks and promises: proxies. Check out this article I wrote on Flippin Awesome (http://flippinawesome.org/2013/11/18/taming-asynchronous-jav...). The following is some sample code to show how code can be simplified with ES6 proxies, making async functions chainable (example in coffeescript):

    getUser = (user) ->
        _("https://npmjs.org/~#{user}")
          .get()
          .extractLinks()
          .filter( -> /package/.test(arguments[0]) )
          .map( -> "https://npmjs.org#{arguments[0]}" )
          .log()
    
    getUser("foo")

where `get` is THE jQuery get for Ajax requests returning a promise and is chained with the rest of the functions in a pipeline fashion. I've found that these techniques can work very well for form validation, as [you can check here](http://www.vittoriozaccaria.net/chained/demo.html) — Firefox required.

Edit: I wrote in markdown but it seems that only verbatim is supported. Edit2: some clarification

The problem with all of these solutions (callbacks, promises, etc.) is that they make you adopt a certain programming style not because it's the most appropriate from a design perspective, but rather to work around scheduling limitations.

Lightweight threads first remove the problem, and then let you choose the most appropriate programming style for your domain. See https://news.ycombinator.com/item?id=7179144

an important omission here is the async.auto function which is a phenomenal relatively new addition that takes a amd approach to asynchronously running multiple tasks at the same time: https://github.com/caolan/async#auto if you haven't seen it already

Here is how I would do it using ToffeeScript (https://github.com/jiangmiao/toffee-script)

    fs = require 'fs'
    async = require 'async'
    path = require 'path' 
     
    module.exports = (dir, cb) ->
      err, files = fs.readdir! dir
      paths = (path.join dir, file for file in files)
      er, stats = async.map! paths, fs.stat
      largest = 0
      for stat, i in stats
        if stat.isFile and stat.size > largest
          fname = files[i]
          largest = stat.size
      cb fname

I use ToffeeScript for everything now, since it makes things much easier to read.

Promises have always been debatable. The new Generators is probably what I am looking for.

Why?

Generators and Promises play very well together.

Why would one use Q promises in Node in 2014? They're very very very slow. Especially when there are alternatives like Bluebird promises that are almost as fast as callbacks.

When we had these things in other languages like Python, node's approach was better. Now that node has adopted them, node is even better still, I guess.

Some other camps ( http://www.erlang.org/ ) still strongly prefer their own approach, and I can't say I blame them.

This is satire, right?

All that work and the findLargest routine doesn't seem to benefit whatsoever from fs.readdir or fs.stat being asynchronous.

Reply on news.ycombinator.com