What it's like building a real website in Node.js | Hacker News Reader

What it's like building a real website in Node.js

travisglines.com

158 pointstravisglines29 comments

Agreed. I spent the last few months building my first Node.js app (and MongoDB) and I resonate with a lot of what's being said here. I wouldn't want to discourage anyone from exploring Node or Mongo, because I think both are amazing projects and have huge potential, but I do think anyone considering either should stop and think carefully before taking the plunge.

Things that stick out in my head:

1. What the author says about the Node community moving fast is absolutely true. Be prepared to spend time upgrading to the latest versions often or deal with dependency hell as you try to ensure you've got matching legacy versions of libraries.

2. Coding async is weird at first, but you may grow to love it (as I have). Over the months, I've discovered many cool tricks and patterns that help keep the nesting to a minimum.

3. Be prepared to fill library gaps, and submit patches if necessary. This means not being afraid to dig into the source code of the libraries you're using. Most libraries have limited documentation so often the only way to figure out how to use them is to get in there and see how they work.

4. Best practices may not apply. In the case of MongoDB, I had to redesign my schema in order to run certain queries now that MongoDB won't support until sometime in the future.

All in all, I'm really glad for the experience and will definitely consider both Node.js and MongoDB for future projects. What will really be interesting is seeing what I miss about Node/Mongo after jumping back into Python/MySQL. :-)

> 2. Coding async is weird at first, but you may grow to love it (as I have). Over the months, I've discovered many cool tricks and patterns that help keep the nesting to a minimum.

Can you share a few? Either here or as a blog post?

If you have a function foo() that takes a callback as an argument i.e. something like:

  foo(..., function (n) {
     // closure over q, r, s
  });

Then what you can do is create a separate function, bar() that takes q, r, s as an argument, and returns a function:

  function bar(q, r, s) {
    return function(n) {
      // still has dependency on q, r, s
    }
  }

Once you have bar(), the foo() call above can be written as:

  foo(..., bar(q, r, s));

and you save one level of nesting. (This also makes the dependencies explicit, which is helpful.)

The main loop of

https://github.com/ithinkihaveacat/node-fishback/blob/ce5f9c...

does quite a lot of this, if you want a real example.

This technique reminds me of Lambda Lifting: http://en.wikipedia.org/wiki/Lambda_lifting

Why not to write it in a much simplier way? Like this:

  function bar(n) {
    // still has dependency on q, r, s
  }
  foo(..., bar);

Yeah, that's nicer if bar() can be defined at the same "level" as foo() but if it makes more sense to move it somewhere else the closed-over variables need to be passed.

We've been using the flow control helpers in async.js to help us with the nesting issue: (https://github.com/caolan/async).

We use the waterfall most, as it allows one to run an array of function in a series with a callback at the end of the series.

This would have been helpful to me when I was starting out building http://domainjig.com - which is a simpler site than the author's. That said, I'd offer these responses to those looking for some advice:

1) Yes, wrapping every database call with an asynchronous callback is a drag. I found a helpful library that made some of my workflow easier to express and maintain: https://github.com/caolan/async And I'm pretty sure I could simplify my code quite a bit more if I understood functional program patterns better already.

2) Libraries can be challenging, but I was able to find good ones for everything I needed. The author discussed version issues. To deal with that, I punted on all those node library managers and just jumped into the library code, checking it in to my repository and freezing it in time (in contrast to the authors point #9 about NPM being awesome for him). The other issue I found with libraries was that there are TOO MANY of them for any single thing. For example, there are at least a half dozen separate libraries to help with asynchronous workflow, and trying to choose which one would be "best" was difficult, and depended on freshness, documentation, and then actually test driving them. Github's encouragement of forking made this extra challenging, with many slightly tweaked variations for each separate library.

I found node.js lived up to its promises (for me): coding in javascript on both server and client was helpful in avoiding context switching inside my head, and encouraged me to invest more effort in writing better quality code everywhere. And node.js is plenty fast.

Another thing I'd add is that I used redis and tokyo cabinet as data stores and found the node.js interfaces to them to be mature and easy enough. It was convenient to use python's interactive shell to create and experiment with redis, and then port the routines to node.js.

To summarize, consider this a positive vote for node.js.

This is a great article but again, like the last article posted here about your MVC setup, I'd personally like to see examples. For instance whats an example of good asynchronous code?

Hey peregrine, I'm working on a solid example here: https://github.com/tglines/throughexample

I'm taking a bit of time with it to make sure its actually useful to people and not just a stub. Feel free to check out what I have up there so far.

Thanks! Just what I was looking for!

Thank you for a good article, Travis.

Quick side note - you might want to add simple validation for sign up form for exipe.com - it currently accepts empty string as a valid value.

If your using Express you might want to look at: https://github.com/1602/express-on-railway

I was hoping to know more about what corner cases he hit where he had to write subjectively ugly code and what part of the ecosystem hasn't matured yet (libraries for certain problems missing, etc). For me, building real applications and solving real problems in any new stack comes down to those two things not being much of an issue.

I remember a friend and I hacking on a RoR app a few years ago where we spent more time trying to get Ruby Gems to play nice rather than actually writing the app. Now, those problems have been mostly shaken out and are a thing of the distant past. Is Node past this productivity immaturity hump yet?

Great article. Writing my node.js project has exposed me to all of the things listed.

One thing I have to say is that NPM is good, but good be better with it's distro integration. The ubuntu install has broken permissions by default, which forces me to install stuff with sudo. I haven't had the time to try and fix it.

I dont think its worth using the Ubuntu install its just too old. At the moment easiest to install by hand, and script something for production.

His issue (though not too obvious) is that NPM assumes /usr/local is writable by unprivileged users, which isn't secure or common in Linux distros or OS X.

It's a bug in NPM, it should be fixed someday.

NPM >0.3.0 changed to requiring sudo while installing. The reason being that it simplifies things logistically as well it allows NPM to install modules as nobody, since you have to be sudo to do that.

I have been running Node.js in production for a while, it is kind of a wild-west, Apache/PHP holds your hand for you and takes care of a bunch of things you might not realize.

If you come from a background of already writing javascript heavy apps, the browser is async so there shouldn't be as steep a learning curve.

Node is moving super fast, with some critical breaks between point releases.

I don't find the community crazy good, but Github is saving the day on node for me - being able to quickly browse through forks is a lifesaver

If you come from a background of already writing javascript heavy apps, the browser is async so there shouldn't be as steep a learning curve.

People say this, but it's not really true. Yes, browsers are async in the sense that your code gets invoked by the browser as events occur. But once your client code gets control, it nearly always proceeds in a completely synchronous fashion. If all you're doing is changing a few dom elements in response to a button click, that happens quickly and all is well, but woe betide if you need to do some long-running computation in the browser... what are you going to do? Build your own event loop with setInterval and chop things up into short-running pieces? That's certainly uncommon. So client-side JS programmers typically don't have much experience with organizing their own code asynchronously, and the learning curve for doing so is still steep.

(There is of course one big exception to what I'm saying, namely communicating with the server via XHR. But not so big that it teaches you how to write a complex program in a non-blocking asynchronous style.)

Web workers are another exception, but I guess they’re not mainstream enough for most developers to have bumped into them.

Access to a client-side SQL database from Javascript in Safari/Chrome is also asynchronous - which takes a bit of getting used to.

What I really mean is JS Apps that use XMLHttpRequest or JSONP -

I found that prepared me pretty well for asynchronous disk reads and asynchronous http requests, I was already doing that in my browser code and it ported quite cleanly to node. (I use a wrapper method that chooses whether to use XMLHttpRequest or http.request based on the environment)

I should note one thing that tripped me up in the beginning is forgetting that I was in a shared environment for my scripts. I typically design server stuff to be 'shared nothing' by default, and you have to be a little careful in node as that is not the default way of running node services and at first I was sometimes sharing objects by accident.

By the way, I created a nice helper to track how forks are going on the GitHub, using RSS feed reader: https://github.com/svetlyak40wt/forkfeed

Great article.

In your opinion, is Node.js mature enough to use in production code ? Or should I wait a little longer ?

Also, what is the sate of the community (i.e. is it easy to find answers and tutorials) ?

I just wrote TLS client certificate authentication for an internal systems API with node, in about 20 lines of code. Node's biggest power is a great abstraction for low level HTTP and socket stuff. You get direct access to everything, with a sane and elegant API.

For plain old websites, I'd still use something like Rails or Django.

I wish we had s-node (s for synchronous) so we do:

  session.start()
  data.load()
  template.render()

instead of the cretan labyrinth it becomes coding for a-node

That completely defeats the point of using Node.js; that is, to use an evented model so your app isn't sitting around waiting for IO. s-node wouldn't be node at all.

FWIW any other runtime/framework will do what you suggest. For example, Rhino.

It would be nice of Node (or rather, v8 underneath it) supported generators and the "yield" keyword like Mozilla's Spidermonkey. Unfortunately, yield is a Javascript 1.7 extension for now, but there's hope for it in ES Harmony.

Anyway, with generators and promises you can write asynchronous code which looks like synchronous code but which under the covers is still completely async and callback-driven. We use it in gjs (a GNOMEy JS runtime built on top of Spidermonkey).

Instead of workflows like:

  function onServerRequest(req, res) {
      db.query("get filename", function(result) {
          fs.readContents(result, function (data) {
              res.write(data);
          });
      });
  }

which can get pretty nasty if you're running things in serial, or want to run a couple of things in parallel but then run something serially, you can have a workflow more like this:

  function onServerRequest(req, res) {
      var queryPromise = db.queryPromise("get filename");
      var filename = yield queryPromise;

      var data = yield db.readContentsPromise(filename);
    
      res.write(data);
  }

The main difference between the two is that the async functions in the latter return promises instead of nothing. The db.queryPromise() function itself would probably look something like:

  function queryPromise(query) {
      var promise = new Promise();
      db.query(query, function(result) {
          promise.putValue(result);
      });
      return promise;
  }

I imagine you could probably write a function to automatically generate promise wrappers for existing async functions as long as they followed a certain schema, and Node's functions tend to do well with that.

Anyway, the onServerRequest() function would itself be wrapped in a function which took the generator values from it and returned its own promise. This way you could stack promises all the way up.

The gjs implementation of this is here: http://git.gnome.org/browse/gjs/tree/modules/promise.js

This is pretty similar to Twisted's inlineCallbacks if you're familiar with them in Python.

Reply on news.ycombinator.com