creationix

How is it binary unfriendly? It's designed to be super easy to parse using C. The lengths and types are in the first byte(s) and most conversions can be done using typecasts. And as far as human readable binary format, it's not that bad. I can usually read a hex dump of msgpack if I've been working with it all day.

Let me weigh in with my experience with msgpack. I'm a long-time nodejs core contributor and have been designing browser libraries for many years. I discovered msgpack almost three years ago and was sad at the lack of javascript support.

Since then I've written and maintain a javascript codec with two optimized versions. One is for [node.js][] and uses node's Buffer methods to do the fast byte to number conversions. The other is for the [browser][] using typed arrays.

I use these libraries in production at various places (the cloud9 IDE backend used them to great effect)

While the native JSON.parse and JSON.stringify is slightly faster than mine with string heavy payloads (and the msgpack is only marginally smaller in that case anyway), when the data is array and number heavy, my codec is much faster than JSON and the data on the wire is a lot smaller.

Also don't underestimate the value of having a binary data type. In my libraries, I extended the format slightly to also encode undefined (as well as null). I also have a string type and a buffer type. In node.js the buffer type is a instance of a node Buffer. In the browser, the buffer is an ArrayBuffer instance (typed array type). So the practical effect is if you put a JS string in, it's encoded as UTF-8 on the wire and comes out the other end as a JS string. If you put a binary buffer in, it comes out the other end as a buffer.

I've contacted the authors of some of the other codecs and when they extend the format, we agree to extend in compatible ways.

My biggest production use of msgpack was as the transport format of my [smith][] rpc system. In smith, an rpc call is done as an array. The first value is the function to call, and the rest are the args. If you're calling an anonymous function, then the identifier is a number. In this usage, the payload tends to be array and number heavy and thus very fast and efficient.

(Edited to add in missing links)

[node.js]: https://github.com/creationix/msgpack-js [browser]: https://github.com/creationix/msgpack-js-browser [smith]: https://github.com/c9/smith

Cloud9 is a very large application. We have many developers working around the clock in different time zones constantly.

The main thing that helps us work together is clearly defining the interfaces between different modules or pieces of the system and keeping things encapsulated.

For example, I work on the VFS interface that other parts of the system use to access workspaces. If I follow semver, then I can push bugfixes without breaking other's code. If I have to change my interface, then I bump the minor version.

Granted, normal npm modules already provide a lot of this. They have semantic versions and the module name is the name to the API interface. We use vanilla npm modules as well. Architect is a layer on top of this.

The real strength of architect over just plain npm dependencies is the config file. You build your architect application as a JSON (or JS) config file that tells the system what plugins to load and what parameters to provide to each. Since the plugins are decoupled and never touch eachother except through defined architect services, it's easy to see if a dependency is missing. You can replace a dependency with some other module that implements the same interface. This makes manual and automated testing much easier. If I want to test my new feature in isolation, I configure a minimal version of the app and add in my plugin. Then I can add more and more plugins as I test to make sure it integrates with everything. I can switch between config profiles with a single flag.

Also having opposing timezones makes it impossible to make real progress on tightly coupled systems. San Francisco is drinking morning coffee when Amsterdam is going home, I'm in between in Red Lick, Texas.

Architect is just a tool that helps in some cases. The important rule is to decouple modules using some tool (vanilla npm, architect, or something else) and then take interfaces seriously. This makes testing sane, distributed development efficient, and large programs possible.

The neat thing about architect is that some plugins provide "services" or named APIs and other plugins consume such named APIs. It's interface oriented programming. The service names are simple short strings and that's why architect has a feature to alias services should name conflicts arise.

Using pure events works too (as does dnode and smith), but for large projects they tend to get hairy. Architect separates concerns into defined interfaces provided by configurable plugin instances. It keeps things clean and configurable.

Like I said in my nodeconf talk. "Callbacks are Hard... in C!" JS callbacks are a very elegant tool for a very hard problem. In node you have the power and responsibility to manually decide when your thread of execution stops and when is resumes. That's what is hard. Coroutines are another tool for the same problem, but they come with their own set of problems and complexities. Callbacks at least are very simple to understand and reason about.

I can tell you the node libraries I wrote for this release are very cool in their own right as well. See the c9 account on github for most of them. I'm especially proud of the vfs system.

Indeed, this is a problem. I've mitigated it somewhat by changing how require works (to be more node-like) and disabling the built-in I/O library. This will make most offending libraries unable to run in Luvit. Also this means I don't get the large existing community of modules either.

Indeed. In that benchmark I did, it was 2 times faster on 32-bit linux and 4 times faster on 64-bit linux. In both the Luvit version used about 40x less ram. Also startup time is much faster in Luvit. This doesn't matter as much for servers, but for embedded devices like writing webOS games for the TouchPad, it makes all the difference in the world!

Node depending on how you build it can take between 300 and 1000 ms of time and 10Mb of ram to just start up on the TouchPad. Luvit starts instantly and uses much less ram.

Larger programs are needed to see how the memory usage and performance scales with real work. That could be completely different.

It's a much leaner VM and it much faster and uses less memory in all my tests. However Lua and JS aren't the same language so real applications may show something different.

Just remember to ask yourself when you're out of hours in the day and still have work to do and family to be with: "What matters most?" http://youtu.be/l70e1TfN34w

As an added bonus, I've discovered that not being able to hack till after the kids are in bed means you're more motivated to get them to bed on time. They get more sleep and you can code once the house is quiet and you brain just had a break. I find it extremely productive.

Also try running apachebench with the keepalive flag. I know it's not realistic, but then again, hello-world benchmarks never are.

I have written extensively on this topic, here are a few articles. Here is the progression leading up to the development of Step.

http://howtonode.org/control-flow http://howtonode.org/control-flow-part-ii http://howtonode.org/control-flow-part-iii http://howtonode.org/do-it-fast http://howtonode.org/step-of-conductor

I never said JavaScript is best language to use server-side. That would be insane. I'm just saying that it worked well for the browser (mostly because it was forced on us, but still) and node is an experiment to try the same thing on the server.

It's a lot better than writing C, (which is what Ryan was doing before starting node) since JS has closures, anonymous functions and other functional niceties that make event based programming much easier.

The fact that you can now code your server-side code in the same language and paradigm as your client-side code is a huge bonus, but was not the reason node was created. V8 is an amazing VM and it's a language that lots of talented developers know. Why not let them loose on the server and see what comes out of this talent.

Given the constraint of using JavaScript on the server, node is the best solution. If you don't want that constraint, then maybe erlang or something with no-shared state is a better solution.

Whether you like the model or not, the fact is that JavaScript with it's callback/event based model is what browsers understand. And the internet isn't going anywhere anytime soon.

Node is an attempt at using this successful model on the server too where is can solve massive scalability issues using the exact paradigm front-end devs already know.

Also while there are many technical ways to make code look blocking, but really be running other events under the hood, it's this exact implicit running of "other stuff" that makes writing threaded code so hard. You have to assume that things can change between every function call because you don't know if somewhere down the chain it's doing pseudo-blocking.

JavaScript's model is simple, you provide callbacks and you know exactly at what boundaries things can happen asynchronously.

In summary, node is one way of doing it. We think it's a better model and it's proven itself in the browser. If you think another model is better, than by all means go for it!

Let the leapfrogging begin ;)

Good catch, I'm looking into this one. I know I've seen this behavior before in some JS environment. I'm glad it's not the case for V8 and will shortly update the article.