Hacker News Reader

Comment by peterhunt | Hacker News Reader

The big reason callback-based systems are hard (despite the fact that built-in flow control constructs need to be re-implemented) is that functions are no longer composable. That is, if I write a bunch of code that doesn't need to do any I/O, I'll just return a value to the caller. If at any point in the future the spec changes and any function in this call stack needs to do any I/O, all the code that ends up calling this function needs to be refactored to use callbacks.

There really should be language support for this sort of thing (like coroutines) so these sort of cascading changes don't need to happen.

View on HN

There are ways to compose functions, like the deferrable pattern. They just all kind of suck and are poor replacements for having a stack.

I'm all for using coroutines to solve this problem. That's the approach taken by my Celluloid::IO library:

https://github.com/tarcieri/celluloid-io

Unfortunately Ryan Dahl is adamantly opposed to coroutines so that's not going to happen in Node any time soon.

I like Kris Zyp's promises and use them a lot:

https://github.com/kriszyp/promised-io

Deferreds make your code nicer, but they still don't magically make your code composable.

What would make Node more attractive is if it supported copy-on-write multithreading and gave me a way to cheat and use asynchronous I/O (like a wait(myFunctionThatTakesACallbackOrDeferred) function)

There are many ways to compose deferrables, primarily by grouping several of them together into something which is also a deferrable. See the em-scenario gem for examples:

https://github.com/athoune/em-scenario

Note: I still think this approach sucks.

V8 provides a really awesome shared-nothing multithreading scheme via web workers. It's just nobody uses them.

Oh right that's definitely true and is much more elegant. I was talking about when a function written in synchronous style (in a long stack of synchronous calls) needs to call something asynchronous.

I wouldn't say the problem with Deferreds is composability - the big problem, as you already mentioned, is fragility towards future changes

Node supports no language extensions that V8 doesn't. Ryan Dahl says:

"Node does not modify the JavaScript runtime. This is for the ECMA committee and V8 to decide. If they add coroutines (which they won't) then we will include it. If they add generators we will include them."

See Tim's thread, My Humble Coroutine Proposal (https://groups.google.com/forum/#!topic/nodejs/HJOyNMKLgB8). Warning: long.

So, you don't like javascript? Don't use it.

But you won't beat the control over program state offered by javascript -- not until computers understand their programmers well enough to reason about program state. We need better computers, better computer science, and better programmers -- that's all. Then we can replace NodeJS with something better.

Until then, NodeJS is almost certainly the most tasteful solution to the most common problems. I hope its replacement meets so high a standard.

> The big reason callback-based systems are hard (despite the fact that built-in flow control constructs need to be re-implemented) is that functions are no longer composable

Absolutely, I've had it happen in "big" (client-side) JS projects.

Only way I've found so far to handle this is to make anything which might ever have any reason to become asynchronous (so anything but helper functions) take callbacks or return deferred objects. Always.

But then an other issue arises: for various reasons, callbacks-based code which works synchronously may fail asynchronously, and the other way around. And then it starts getting real fun as you still have to go through all your (supposedly) carefully constructed callbacks-based code to find out what reason it would have to fail (alternatively, you create both sync and async tests for all pieces of callbacks-based code)

Coroutines on a server...like OpenResty [1] (a package containing Nginx w/Lua integration)?

The example "asynchronous get" becomes something like:

    local myObject = query{ id=3244 };

The query function can be written to handle a cache lookup, a database query that gets stored in the cache, and any other logging you want, because Lua has coroutines. All I/O that gets sent through the "ngx" query object (which can connect to local or remote ports) yields control to the main loop.

"query" is an example of a function you could create; its implementation (with a cache lookup) could look something like (yes, I use CouchDB...):

    function query(t)

      local result = ngx.location.capture(  "/cacheserver/id:"..t.id );
      if #result == 0 then
        result = ngx.location.capture(  "/couchdb/usertable/"..t.id );
      end

      return result
    end

I've heard reports of 50k+ connections/second on a VPS running Nginx, LuaJit, and the LuaNginxModule, and on my low-end VPS it easily handles 2000+ connections per second (with CouchDB queries) with no more than 250ms latency. Actually, that was as many connections I could send at it, so it may be able to handle a lot more.

[1] http://openresty.org/

I'm guessing monads will be a potential solution implemented pretty soon. I'd be interested in seeing how that works out.

There is already a Monad based solution - its Deferreds / Promises.

However, monads don't solve this problem - they cause it, since their primary concern is correctness and not converting between monadic and non-monadic code. If you have a pure function and need to convert it to a monadic action there will be lots of collateral damage as functions that interacted with the old function have to be converted to monadic style.

Yeah, just after I wrote that I realized that Deferreds/Promises are monads.

You don't have to convert them, but you do need a way to write code using those pure functions "inside" the monad which I don't think is easy with this model.

You should be able to just chain your actions together, with a wrapper function to turn pure functions into actions. I don't see why you would need to convert the actual pure functions to monadic style.

I guess what I had in my head when I made the comment was the convenience functions or the syntactic sugar around a monadic solution. Seems like the node guys are playing with things like this but all the solutions I've seen seem so ugly.

Pure code can be used in the monad just fine, (ie.: you can return values up to the monad and lift functions)

    //turn a function into a monadic action
    function lift(f){ return function(/**/){
        var d = new Deferred(); //Im using the Dojo API
        d.resolve( f.call(this, arguments) );
        return d.promise;
    };};
    
    var f = function(x){ return x+1}
    
    lift(f)(0)
    .then(function(y){ ... })

The problem is the opposite direction: pure code using monadic code and pure code turning into monadic code

If you have something like

    var x = f1( f2( f3() ) )

And f2 becomes a monadic action you have to rewrite this bit as

    var xPromise = f2( f3() )
    .then(function(f2result){
        return f1(f2result);
    });

ah, very nice, it is easy. The chaining is still fairly clunky though, although maybe comparing it to haskell's do notation syntactic sugar is unfair.

(isn't the opposite direction the feature of doing things this way? That you can't call monadic code from pure code is a good thing)

Callback-centric code can be composable if you adhere to the continuation passing style (CPS) discipline. While it can be difficult to write CPS code directly, it's a well-trodden path in computer science.

And the people who trod that path all came back and told us that it's a worse way for humans to write and reason about code.

Not that it can't be used under the hood of course.

In this sense, I think the approach of extending Javascript with async features and compiling down to CPS (ala http://tamejs.org/) has a lot of promise.

Yeah that's true, but using CPS for everything means you can't really use normal flow control anywhere which IMO makes it inappropriate to use for day-to-day engineering.

I have been using node.js for my day-to-day engineering for the past year. It certainly takes a couple of weeks to get a good feel for solving problems without being able to use normal flow control. But once you get it becomes easy. So I wouldn't call in inappropriate, just different. It's much like the feeling of learning your first functional programming language. It's not wrong, just different until you adjust the way you reason about your programs.

CPS is also generally used as an IR in compilers, not something you write by hand

Yes. In Scheme, call/cc is generally not used directly. Instead, you build higher level abstractions, for example full co-routines or perhaps more limited "generators" like Python has. Co-routines have been around for more than 30 years. It's quite said to see the Node.js developers ignore decades of language research. Implementing concurrency using callback functions is definitely a "worse is better" approach. Maybe it will win because of that.

Edit: I guess a hopeful idea is that there should be no reason something like call/cc could not be added to Node.js. In that case, the extensive library of non-blocking functionally will be very handy since you could build a sane continuation based interface on top of it and escape from callback purgatory.

Node has coroutine support via the fibers package: https://github.com/laverdet/node-fibers. For some benchmarks, check out my Common Node project's README: https://github.com/olegp/common-node

Reply on news.ycombinator.com