Node.js is genuinely exciting | Hacker News Reader

Node.js is genuinely exciting

simonwillison.net

197 pointssimonw67 comments

I just jumped on node a few days ago.

People who are already comfortable with programming with events and closures in Javascript are going to try Node and say things like "where has this been all my life?". In those few short days Node has highlighted to me just how dramatic of an impact a tool that is close to your language/model of choice can have on your ability to rapidly turn ideas into working code. For a cross section of coders that work with jQuery all day, Node will be a big deal.

Node has some really friendly and highly productive people working behind it, chief among them Ryan Dahl ( http://github.com/ry ). To learn Node's ins and outs and have some fun, I spent this weekend building out a simple node-based MapReduce-like work unit processor and needed a lot of random help along the way. When I needed gzip ( http://github.com/waveto/node-compress ), or to communicate with Unix command processes, or a redis client ( http://github.com/fictorial/redis-node-client/ ), the Node community was there to help me very quickly.

Simply said, Node.js is a very productive and fun environment for weekend hacks, and I expect it to become a primary fixture in my toolbox in the coming months.

[edit: added links]

It's really awkward to write evened code without a good implementation of futures ( or go's channels ). The problem is that you end up nesting a lot. Look at the example for the posix rename here: http://nodejs.org/api.html#_posix_module

Now imagine implementing a normal http controller. You have to do say 5 MySQL call (or redis, whatever) to get all the data you want. This means that you have to nest together 5 closures. Now if any of the methods fails and there is no global error handler that disconnects the client then that means that the connection will hang forever until the browser user hits the stop button. Spotting these situations becomes really hard with complex code.

Futures ( Go channels ) solve this because they block the thread when someone accesses the data. If you want to do 3 db calls you can just write familiar code:

var a = db.select(...); var b = db.select(...); var c = db.select(...);

print c #=> blocks until c has returned

"The problem is that you end up nesting a lot."

I haven't done that much of asynchronous programming, but I think you end up creating some helper constructs that help prevent excessive nesting. For example you could create a function that takes an array of functions and calls them sequentially (in the sense that when the first callback is executed, it calls the second function and so on).

You can't get around nesting if functions take a callback in their interface. It's viral. You can mitigate _some_ of these problems by defining a custom event system, but this makes code logic very hard to follow. It's better in a complex system to hide the mechanism altogether. A little higher on the thread I point out that this can be done by custom events and function decoration.

This is precisely my concern, and I have talked to a lot of people regarding this exact point. I think node.js is really cool but I would love if this was just built into the language, such that code would perform asynchronously but not move to the next line.

var a = something_async(), b = something_else_async();

Upon calling something_async, the code starts waiting and blocks further execution until it is ready, going to the next statement. This is not a foreign concept at all, and is used in javascript as well with alert, prompt, confirm:

fries = window.confirm("Do you want fries with that?")

confirm is an async method (has to wait for the user), but the next line is not executed until it returns. I personally see no advantage to creating a system of callbacks:

do_x(function () { do_y(function() { do_z() }) })

when this would be much more readable:

do_x(); do_y(); do_z();

and would have the exact same perf characteristics. In fact, you could probably simpy code transform the latter into the former (I know this is what stratified JS does), although this is probably the worst way to do it. What you really want is to just auto-call "wait" on every promise (I believe wait still gives you the asynchronous benefits).

Of course, the main "drawback" to this is that you need language support to pull it off, so you need to implement a preprocessor of some sort.

I don't understand your concern. When you say 'Upon calling something_async, the code starts waiting and blocks further execution until it is ready', you're describing a synchronous call. Indeed window.confirm is a synchronous function--nothing is executed until it returns.

And yes, both asynchronous and synchronous operations have their place, umm, except perhaps in Haskell where things get funky. Chaining async operations together in javascript can be tedious, but its easy enough to write a helper function to handle it for you. Every javascript app I've written that uses AJAX always has some form of createCallback(context, function, arguments) function.

Re-read my example. All 3 db queries are run in parallel however the code does not continue if you access one of the values until the value is available. In a typical web application you usually get 3-5 things from the db, then you start rendering the templates. Each DB query waits for completion, then it starts the next which is a huge waste of time. In fact the system can easily start the parser and template renderer while the DB queries run and all the data will likely be available once it's accessed. This means you have all the benefits of synchronous processing with the majority of the advantages of async. The vast majority of web applications would likely very rarely block a thread in such an architecture.

I'm very surprised that not more languages implement this pattern. Maybe i'm missing something but this is the major feature of GO, except that almost everyone missed it and focused on the fact that it compiles fast and looks like c

Seems like you the dataflow paradigm is exactly what you'd need. With dataflow, trying to read a variable that doesn't know its value yet blocks until the value becomes known.

You don't need language support and you don't even need a preprocessor. You just need to define a custom event system.

  do_x();
  do_y();
  do_z();

Ideally those three should not even be dependent on each other (FP to the rescue). They should absolutely not block if you're going to write your code this way. If they are dependent and need to block they should be written this way:

  do_z(do_y(do_x()));

Each one delivers a promise for it's computation. All you need is custom events and function decorators. Port my library Promises.js to Node.js ;) It's only about 300 LOC (another 5 or so to implement function decoration).

http://github.com/ShiftSpace/promises

While not in the example that you replied to, it's assumed that you need the return value and want to do something with it. Think database query.

precisely, ideally this should just work:

var result = do_x() + do_y();

and could. Again, if I understand the documentation for node.js correctly, simply automatically appending a .wait() to every promise returned should in theory give you this behavior (and be as performance/non-blocking/etc) as the other method.

  var resultp = add(do_x(), do_y();
  do_z(resultp);
  do_foo();

You're code will block on add. This code only blocks on unrealized promises, that is do_foo gets to execute. This is _more_ concurrent.

You're no longer arguing your original point, or at least, you're no longer arguing against me. Your code and my code are now nearly identical, except for the fact that you use an add function and I use an add operator. In fact, if I just say "my add operator can manipulate promises" or whatever, then they become 100% the same. Since my original point was that I'd like to use a preprocessor to make asynchronous programming as easy to transition to as possible, this would be very easy to achieve (just replace exp1 + exp2 with add(exp1, exp2)). That's what I meant by "just work", not "almost work". The goal was to not have to use callbacks and nesting, a point that both our code snippets have reached, so I'm not sure exactly what we're arguing about.

Heh, we basically understand each other now, a couple more illustrative examples always helps point that out. And yes a preprocessor to convert operators to functions would be quite useful.

I'm excited about it. I've been attempting to blog, and using node.js as my subject matter. (Don't want to link-pimp as I need to enhance/edit, but you can check my profile if you're interested).

It's my opinion that JS is the new lingua franca of programming, and it's about time we replaced C/Java. It has consistent syntax, some great functional aspects, dynamic typing, and it's now become a top target for optimization by some of the world's smartest engineers.

Breaking into systems coding is the next step, IMO, in JS becoming a legitimate option for "anything" programming. I think node.js is going to be the library for it. It's really idiomatic and concise, and it's got just the right amount of opinionatedness I think to become the systems and JS version of Rails.

> It's my opinion that JS is the new lingua franca of programming

Indeed. And its new incarnation, ECMAScript 5, fixes many problems with the existing language and makes it a lot more powerful and better suitable for programming "in the large".

> It's my opinion that JS is the new lingua franca of programming, and it's about time we replaced C/Java.

Steve Yegge beat you to it: http://steve-yegge.blogspot.com/2007/02/next-big-language.ht...

I think I've read a couple of perspectives on why JS would be the "NBL", and I wasn't trying to claim I was the first to come to that conclusion.

I'm saying that I think that this is JavaScript's first implementation in the wild that I've liked enough to want to do non-web programming in.

I thought doing "http server programming" is pretty much in the same category as doing "web programming"?

This article is about HTTP server programming, but node.js implements a number of features that are just good for general-purpose programming of all kinds: a good system for including modules, a decent I/O interface, robust events.

I'm not so excited by this guy writing an HTTP server in JavaScript. I'm excited about the potential of JavaScript in writing elegant, high-performance apps that aren't tied to the web.

Learning Scala right now. Seems it fits nicely as the NBL according to Steve Yegge's criteria:

#1 C-like syntax (might be closer to Ruby actually)

#2 Static-typing but feels pretty dynamic due to extremely good type inference

#3 Perform on par with plain Java

#4 IDE(Eclipse/NetBean) support

#5 Everything in the "Kitch sink" section included + Actor-based threadless concurrency + easy to write embedded DSL + (I heard even continuation is coming soon). So what's missing is a macro system ...

#6 JVM so multiplatform by default (even on Android)

I code in Scala for some time now. Love this language! Though I think that Node.JS is something different and has different purpose(yet to be discovered). Scala as any language that runs on JVM, doesn't completely support Non-Blocking IO(this should be fixed in Java 7 though).

"JVM, doesn't completely support Non-Blocking IO"

Isn't java.nio supposed to provide non-blocking IO (at least for socket)?

Yeah, for sockets java.nio does that just fine, not for files. Of course, it's an improvement over plain java.io but still it's not non-blocking.

Actually, given improvements in threading, NIO doesn't give faster socket throughput in typical many-connection servers. See:

http://paultyma.blogspot.com/2008/03/writing-java-multithrea...

JVM so multiplatform by default (even on Android)

Even on Android what? Which Scala based project I could run on Android unmodified?

Where I can get JVM for linux/bsd on ARM?

What exactly is the driving force behind the desire to do everything in Javascript? Is it just because we have a surplus of Javascript programmers? It doesn't seem to me that this task is particularly well suited to Javascript. (I think) Something like Erlang would be more applicable.

And to me it seems that learning a new programming language (even one using a relatively obscure syntax), is a much simpler task than designing an entire project (or piece of a project), around a language that doesn’t really fit the needs of a project.

To that extent, I think that the idea of a ‘lingua franca’ language is an inherently broken metaphor. Programming languages are closer to human-computer interaction tools than they are natural languages. ‘Lingua franca’ implies that every language can communicate approximately the same thing (it is just a matter of who has influence). I think this is a mistake, as different languages allow you to think in hugely different ways about programming. Different languages have different intrinsic concepts.

It may be that this argument boils down to ‘if all you have is a hammer, everything looks like a nail,’ but even that is an oversimplification, as I don’t even use the same hammer for all types of nails (tack hammer, framing hammer, etc).

I am not saying that node.js is necessarily a bad thing to have. It may turn out that Javascript is indeed particularly well suited to this type of application. Superficially, however, nothing convinces me that this is some sort of exciting home-run-win situation (or that it is any better than any other functional-style programming language).

Could you explain the excitement and appeal to me?

I agree. This language is not a general-purpose programming language, at least not at the moment. Although, I can see how I can use it to build a small(but customized to my needs) COMET server in a more simpler way.

I think there is definitely a communicative advantage to the 'lingua franca' concept as it applies to programming languages. Think of programs as the semi-English (English is actually the true lingua franca of programming :)) representation of Turing machines. In this sense, every language can communicate approximately the same thing.

I don't think JavaScript will cover every paradigm in the sense of obscure language concepts, in that it will likely never have Lisp's complicated macro system or Ruby's ability to make really fluid DSLs. I don't want it to; the appeal to me is that what's going on in JavaScript is almost always self-evident (if you avoid the bad parts). () means you're calling a function or forcing operator association. {} means you're defining a block, array, or dictionary.

I do think, though, that it could establish the new minimum standards of the 'demo language' for an algorithm. For algorithm demonstration and sharing, I think it's absolutely awesome that there is a language that is now capable of doing anything you want with an interpreter (albeit DOM-confined) pre-installed on every single computer you use. I don't want to read Wikipedia algorithms in some hacked-up Pascal. Give me something I can run now, but that's universally understood: a strong common medium to explain programming languages in unobfuscated terms.

"Could you explain the excitement and appeal to me?"

There's not much I can say that I didn't already say in my blog entry. JavaScript is designed for event-driven programming in the browser, and my brief experience with node.js makes me believe that it also works well for event-driven programming on the server. And V8 is really fast.

As for Erlang... when you're programming for the web, it's useful to have a programming language that treats strings as more than just an array of integers.

With regard to Erlang, are you sure? Joe Armstrong seems to think its fine for string manipulation...

http://www.erlang.org/cgi-bin/ezmlm-cgi?4:mss:47293:200911:d...

(You can either think that he is biased or that he would be an expert on it).

I mean, it seems to me that javascripts main advantage in programming for the web would be DOM manipulation. And that is good, but a lot of languages can do it. (Honestly I've never really done (much) direct string manipulation for web programming).

It doesn't seem to me that javascript really has a monopoly on event driven programming either. It is relatively easy to do that sort of thing in almost any other functional style language...

While not perfect, JS is actually pretty pleasant to use. Also I like not having to use different languages between client and server.

What popular languages are cooler than JS? Maybe Ruby, but for my taste there is too much metaprogramming in Ruby which makes it very hard to master. I like the extreme simpleness of JS (basically everything is a hash, that's it).

Though I must admit that now that it looks like it has a shot at being the language for everything, I am getting second thoughts. Those brackets can get ugly - something with a slightly cleaner syntax might have been preferable. But all current languages have their pros and cons, and JS doesn't fare badly in copmparison.

For example I think Ruby takes it too far - you can get clean syntax, but the tendency to create something like a DSL for everything is also a burden for learning. Plus, Ruby is still too slow for a lot of tasks.

I'm not sure about lingua franca but it does kinda want to be the language of the web. You're already running it on the browser. So there is a synergy in having it on the server also. GWT lets you do write sides in Java, but there really isn't any reason not to do both sides in Javascript.

I loathe the javascript syntax but am sold on this lingua franca thing, although probably for reasons different to the person who coined the phrase.

    Could you explain the excitement and appeal to me?

Yup. Javascript combines several elements that give it a reach for high power.

* Javascript is a prototype-based language. I find this to be a better way to write software than other approaches.

* There's a good ometa implementation for javascript. Ometa is a powerful ways of writing domain specific languages.

* Modern JVMs support javascript, meaning that anything you do in javascript can be adapter to leverage existing Java codebases.

* Combination of javascript on both the client and server will likely expose new possibilities. For example, I've been thinking about the possibility of a 'builder' app in javascript whereby you use a firebug console to dynamically build a form and attach business logic to it and then freeze it back to the server. Yes, you could do it at the moment, but it's less effort if you only have one codebase to think about. The thread of bloat in the client becomes less of a problem with new approaches to development like Google Gears, also.

Node.js is an important foundation to javascript establishing itself as a viable server-side language. At a glance, it looks to be a good balance of what's needed, also.

> Twisted and EventMachine are hampered by the existence of a large number of blocking libraries for their respective languages. Part of the difficulty in learning those technologies is understanding which Python or Ruby libraries you can use and which ones you have to avoid.

This is what's interesting, I think. The reason Erlang does so well at this is that stuff written in Erlang basically just doesn't block because a scheduler sits under the whole thing.

> Node creator Ryan Dahl has a stated aim for Node to never provide a blocking API—even filesystem access and DNS lookups are catered for with non-blocking callback based APIs.

So that's the tricky bit, I suppose, especially as your language/environment grow, and people start doing random stuff with it.

BTW, it's also worth mentioning that Tcl has had a nice event system since well before Twisted came out, built into the language. However, like Python, you can't count on random commands not to block, so you have to be careful.

I don't find that particularly true with Twisted because for every common blocking library there is a Twisted equivalent. Whether you need http, ssh, db, or a multitude of other protocols and interfaces, Twisted has them. Sure, you have to be a little careful, but chances are if you're using an event-based networking engine you have a good reason for doing so and a pretty specific purpose.

I've compiled it and had a brief look at this. I have got only this to say, it is a very important evolutionary step in server-side coding and coding in general. Really impressed! Though I have one more question, how do you handle concurrency? I mean, mutex, locks, etc. Don't get me wrong, I'm not being picky, just wondering what could be the best solution for this.

Edit: Or Events in EventEmitter are being executed one at a time, so there is no need for locks?

Node is single threaded, so you shouldn't need to implement any locks yourself.

(Comment repeated from proggit, where this was also posted.)

Don't get me wrong here, I love coding in Javascript and have often entertained the idea that server-side Javascript would be fun. I also totally buy the need to get away from thread-per-request for many web serving scenarios, Comet being a perfect example.

However when I see someone going so far with this as to say "all APIs must be non-blocking" I have to wonder if they would be better served by using a language where threads were less expensive. I know closures make this kind of thing a lot less of a pain than back in the day when we were programming against OSs with no threads and async APIs were often a necessity, but to mandate this style is basically asking developers to manually rewrite all of their logic into continuation passing style. It's a bit, well... naive is the kindest word that comes to mind.

BTW, I don't think it's a coincidence that all of these event-driven web frameworks are for interpreted languages with crappy thread support.

The 'Hello World' example (at the top of the front page of the Node.js website) is a server that listens to http requests, waits 2 seconds, then responds 'Hello World'.

The server is supposed to continue responding to incoming requests even while it is waiting 2 seconds for previous requests. So I figured that if I quickly queued up a bunch of requests (say, over the course of one second), they would all respond just as quickly after the two seconds was up.

However, each response came a full 2 seconds after the previous response. I increased the wait argument to 10 seconds just to be sure.

This seems functionally equivalent to blocking. Is this the way the example is supposed to work?

Was your client sending the requests concurrently? If I do that and run "ab -n 100 -c 100" (for 100 requests run with concurrency 100) the benchmark takes almost exactly 2 seconds to execute all 100 requests.

I guess not - it worked when I followed your suggestion. Thanks!

I was naively cranking out a bunch of tabs in Firefox really quickly. :) I guess Firefox doesn't make concurrent requests when it's the same URL...?

Firefox only makes two (by now maybe four) concurrent requests to the same server as mandated by the HTTP protocol.

It's technically not mandated, but recommended. Servers need to be able to deal with more anyway, as proxy servers are recommended to open up to 2 * [no of clients] connections to upstream servers.

Firefox has a maximum of 2 connections per host - though it's being upped to 6 in Firefox 3.5.

It is unfortunate that none of the JS VMs have support for concurrency. Event driven programming can get messy (lots of state machines).

It's usually pretty trivial to create an event bus where you can publish and subscribe to events.

How is that equivalent to new threads (stacks) of execution?

It's nothing to do with concurrency - the event bus pattern will alleviate much of the pain around event-driven programming (the FSM's you were referring to) and decouple the event handling machines. In my experience, moving to an event bus model makes event driven programming very manageable.

If the event bus has executors (which it could in v8), it will be parallel to boot.

Or tail-recursion optimization, for that matter.

Rhino does, but it's an order of magnitude slower than SquirrelFish/JavaScriptCore and V8.

"Exciting" is an understatement. Cloned and compiled it yesterday and since then I can't bring myself to stop playing with it.

I also recently cloned and compiled it myself, but I haven't had much time to play with it yet.

I have been doing a lot of Javascript/jQuery UI stuff at work lately and have become VERY comfortable with the anonymous callbacks and such. Now I'm ready to jump on Node and see what I can do. This looks awesome!

Sorry, can somebody explain me how this works? Obviously, I'm missing something. Is the code being interpreted on the server-side? Then my question is, how fast it is and if it's capable to serve large number of connections? Actually, it's very exciting if it runs no slower than Java.

Yes, it's being executed on the server. It's very fast - 3,500 requests a second for returning a "hello world" string with 100 simultaneous connections, and because its all event based it should be able of handling thousands of concurrent connections without much trouble (I haven't tried yet because my laptop runs out of file descriptors).

You know you can just bump allowed file descriptors up with ulimit, right?

Glad I read this. I'd assumed Node.js had something to do with the Nodebox visualization environment on Javascript (I think a project like that has been featured here before). While that would be interesting and pretty, it wouldn't be my cup of tea right now.

This, however, looks really cool. I've been waiting for a compelling JS server side framework for awhile—definitely going to have to check it out. I hope other people aren't making the same mistake.

For those of you who have taken a look at this, is there any library or construct in python that would be equivalent to the Events in Node.js? http://nodejs.org/api.html#_events

That's the deal maker for me... To have custom defined events. Correct me if I'm wrong but even Twisted doesn't have this facility?

I'm pretty sure Twisted's deferreds serve the same purpose.

It's great to see node.js starting to support CommonJS (http://commonjs.org/) modules.

Very exciting! What sort of persistent storage options are there for something like this?

There are several options already available, including file i/o and Postgres db connections. cf. http://wiki.github.com/ry/node

What's the main one advantage of Node.js over php (or ruby)?

As I said in the article, it's event based which means it can handle massive concurrency and serve pages that don't necessarily reply straight away - perfect for implementing things like comet. It gives you similar capabilities to Ruby's EventMachine. I don't think PHP has an equivalent at the moment.

To me, the important part of node.js is that it's using the V8 (http://code.google.com/p/v8/) JavaScript engine which Google has a strong, vested interest in being the fastest dynamic platform around and a lot of really smart programmers to make that happen.

Otherwise, it's a matter of preference. What's the main advantage of PHP/Ruby over Perl?

Does v8 know supports cross-platform aio, epoll, kqueue or at least sendfile()?

Node uses libev: http://software.schmorp.de/pkg/libev.html

Reply on news.ycombinator.com