Messenger.js - dead simple API for multi-server communication in Node.js | Hacker News Reader

Messenger.js - dead simple API for multi-server communication in Node.js

github.com

68 pointsweixiyen26 comments

One of the things that RabbitMQ does which this doesn't seems to be much more support for variable topography. This seems to need to know of all receivers when you create a "speaker" which in dynamic networks doesn't seem like such a good idea. Of course you can have each receiver register with some sort of registry and have speakers interrogate that, but then you're back with a single point of (logical) failure.

Discovery and dynamic usage doesn't seem to be covered in many of these projects (cool as they are) which seems to me to be a more challenging and useful extension.

It's a very good point. I plan to add automatic discovery where every speaker and listener maintains a current list of the communication topography.

I figure I'll use this space to explain "why" messenger.js even exists.

One reason I wrote messenger.js is because I find it easy to install, work with, and deploy over using RabbitMQ in node. Personally I was using zeromq with node-zeromq.

This means, I need to install zmq, install the node module for it, and anything else that it may need to work properly. In addition I have to do this on dev and production.

On top of that, I also have to write an abstraction over it similar to how the messenger.js listener works right now. To me, there seems to be too many moving parts to the puzzle just to get some simple communication going.

I fell in love with the DNode API at one point, but I have to start my servers first and clients second. It doesn't appear to be a way to capture error events and try to auto-reconnect. There is a ticket for it but it seems neglected. For me, I don't want to worry about that. I want that abstracted away. I also wanted it in TCP, not HTTP.

Of course, the benefit of RabbitMQ is in a separate language, someone probably already wrote the adapter for you to communicate with services of different languages.

I'm not trying to compete with RabbitMQ here though. Just trying to get node.js communication to work in a scalable fashion with a 1 line install that just works.

Yeah that's fair enough - I'm working with RabbitMQ in Node at the moment, and I've got a simple domain specific wrapper around it, which I find fairly easy. Of course, not everything on it is in Node, which is one reason this wouldn't work for me - but if it works for you, that's great!

I'll be keeping an eye on it anyway - you never know when something like that might solve a problem you encounter.

The API for auto-discovery will simply be the same API but pass in namespaces instead of ports / addresses.

This looks promising. Trying to figure out how to benchmark this to test the "insanely fast" claim though. Any ideas? Maybe even add a tiny infographic on there?

Absolutely. Such a claim needs a serious backup in the domain of messaging, especially when the competition includes fast and reliable solutions such as Apollo or RabbitMQ.

How does this compare to AMPQ solutions? This definitely looks like a great use of Node though.

What makes you think this is faster than existing solutions? I see no benchmarks.

There's no need to benchmark. Everything written in Node.js is insanely fast by default, because nonblocking is the secret to the roflscale sauce.

I will put them up. In theory it should be faster than something like DNode or anything that uses DNode (hook.io) because it doesn't have the TCP tear down from using HTTP to carry what is probably small payloads most of the time.

Right now it's as fast as Node.js will allow TCP to be, if that makes any sense.

Benchmarks will come as soon as I get some free time to sit down and do them all. I'll plan to compare with zmq+nodezmq, rabbitmq+nodeamqp, dnode, hook.io. Any others?

You do know what HTTP keepalive is, right?

Eh, correct, but you still pass unnecessary headers each time as part of the payload. It would even up the speed theoretically, but it's still a request reply model that demands a response, and you really don't need that for pub/sub for example.

TCP just feels cleaner as a protocol for this kind of stuff but I would certainly concede that speed of HTTP isn't as bad as people would make it out to be.

Also, I'm considering adding createFastSpeaker and createFastListener to the API for people like game makers who don't need to guarantee delivery of certain data over UDP.

While it's nice to see projects such as this, it's somewhat of a pity that most of such solutions are specific to a single language, runtime or environment. I recently started working on something similar, but which is supposed to glue code from any language or environment together (somewhat like AMPQ etc, but on a slightly lower level, I guess...): http://dev.yorhel.nl/doc/commvis

Kind of like 0MQ, but with JSON?

Ah, I hadn't heard about 0MQ yet. Just glanced at it and it looks quite similar, but with very different communication primitives.

I certainly need to play with that to find out more. Thanks for the mention!

np!

So, not to be dense, but could you give an example of an application that would make good use of this library?

It feels very websocketish, but with some more infrastructure around it. Would people use this instead of something like socket.io, or are the 2 solving different problems?

Looks pretty cool, either way.

I think it's more like Hook.io than Socket.io. One use could be server monitoring or deployment. You can also check out the Hook.io hook libraries to see how that has been used: https://github.com/hookio/hook.io/wiki/Hook.io-Libraries

This seems like a simpler API to me.

Thanks for the reply, I'll check out Hook as well.

I'm fairly new to working with Node, so any kind of context for use-cases around some of these libraries is always a help

i assume messenger works the same way, but the nature of hook is the processes are completely decoupled. essentially, you are just creating an event emitter that works cross-process. one benefit of this decoupling is it enables you to spin up and take down these processes without affecting the rest of the program.

Personally, I have a scraper service, web app server cluster, stream server cluster, mobile service, and a data access layer in my setup.

They all communicate with each other, but in very different ways.

The scraper for example uses fire and forget to send something to the data access layer.

The web app cluster needs to use the request reply model with the data access layer.

The data access layer needs to send pub / sub messages to the stream clusters holding live connections.

I also have a mobile service which does a combination of pub / sub (notify stream servers), req / rep (data access), and fire / forget (analytics).

I needed one library with a 1 line install that handled all of these cases, with the simplest API I could come up with.

I originally built node-monitor (https://github.com/franklovecchio/node-monitor/tree/v0.5) to monitor multiple servers; I was trying to emulate Splunk by sending all data from an environment to a couple servers, which then pushed out to websockets (live log viewing) + POSTed to CloudWatch. Ended up throwing out the socket piece because it was a pain in the ass to manage.

On a side note, you can scale what would be single-instance architecture with something like messenger (think an MQtt.js broker).

The examples and output don't seem to be synchronized (for example in the pub/sub example). But more interesting, how can the client reach servers on a different network/ip? since that is the more common case for distributed servers...

messenger.createSpeaker('22.222.22.22:2222', '33.333.333.3:2000')

on other server

messenger.createListener(2222, 2000)

You'll of course have to update your IP Tables.

Oh no, I don't think the internet can take another 'so fast it destroys everything you know about interneting' node.js project :(

similarly, check out hook-io by nodejitsu. i actually don't see the benefit of messenger.js at all in comparison. perhaps its simplicity, if that happens to be beneficial in your scenario.

Reply on news.ycombinator.com