ZeroMQ: Modern & Fast Networking Stack
igvita.com
igvita.com
There's some built-in support for common topologies (client/server, publish/subscribe, pipeline), and if you're willing to use the undocumented XREQ/XREP socket types, you can build more exotic topologies.
Most of the value in ZeroMQ is the actual C++ implementation, so other languages generally just wrap the C++ code. The zeromq API is tiny.
I haven't used it on a big production project yet, but I hope to do that soon.
I want to correct that assumption right now - it's one of those places that higher level networking libraries often fall down - because simply 'fire and forget' is a leaky abstraction.
Let's say you're streaming a large file over TCP to a client. Generally, disk reads are far faster than you can send data over the internet. If you naively keep sending data, you'll quickly fill up the OS buffers - and begin to consume memory as it's queued on the client side. In an extreme case, if you're sending a multi-GB file, you'll probably end up with most of the file sitting in RAM (assuming you don't run of ram and start thrashing). And for what? You really only need to load additional data in from the file as fast as you can send it in the socket.
In a real-life case, streaming multiple files to multiple clients, you'll quickly run out of RAM with even moderate size files.
Some high level libraries provide an event to let you know that the OS buffers have been emptied out. For example, Node.js has a drain event. But this has to be handled explicitly, and isn't really highlighted in the docs. But if you write your code naively, you will definitely run into problems.
While it's great that many of these high level networking libraries have tried to abstract away many of the complexities of network programming, they're also a trap for the unwary. Just remember that they aren't a silver bullet that let you handle sending data over the internet as a simple function call.
It's hard for me to tell if your criticisms are based on real-world experience with the library, a review of the available documentation, or just my description of a single feature intended to reduce the cost traversing the userspace/kernel barrier.
I'm definitely interested in real-world experience with ZeroMQ and any problems you've encountered. We may begin some larger-scale testing soon, and would love to know about any problems.
It's also wholly unsuitable as a floor wax or a dessert topping. :)
Does your criticism also apply to its intended use case? At a glance, it doesn't strike me as an unreasonable design choice, but I haven't thought about it deeply.
Is there like a library or a protocol that is tuned towards transferring multi-gb files ?
For simultaneous large files with no special needs, my instinct would be to just choose any lightweight HTTP server that supports sendfile(). Basically, pass it off to the kernel, and let it handle it. But maybe there are more subtle keys to good large file performance: http://www.psc.edu/networking/projects/tcptune/
For cases where you are controlling both ends, you might find useful suggestions here: http://serverfault.com/questions/111813/what-is-the-best-way... In particular, Tsunami and uftp look interesting.
Try it, you'll be surprised, I guarantee it.
This lets us do things like start clients and THEN start a server... the clients automatically connect when the server comes along. The server can go away, be replaced by another on the same endpoint, and clients will gracefully and invisibly start talking to the new server.
In some cases this is precisely what we want, in other cases it's not. If we need to detect dead peers, we add heartbeating as a message flow on top of 0MQ. Most larger 0MQ applications currently do this. Eventually 0MQ sockets may offer heartbeating, it seems a natural evolution. ("Seems" but is not necessarily.)
Additionally there are some patterns (like synchronous request-reply) that simply don't handle network issues properly. If your service dies while processing a request, your client if it does a blocking read will hang forever. There are work-arounds such as using non-blocking reads.
It would be a mistake to try to solve all challenges at once in any project. 0MQ is taking it step by step, starting with the core questions of how to design massively-scalable patterns correctly. Just doing that is worth gold, as you can see from people actually using 0MQ, who consistently tell us, "this makes our life orders of magnitude easier, thank you".
And as we solve the core problems properly, we'll continue to address other aspects, either in 0MQ core or in layers on top of that, and this process will continue ad infinitum.
Edit:
1) It's a networking library; no admin tools or other soft handle-bars, like user-space utilities.
2) It uses a binary protocol. Good luck debugging that with syslog.
It's a very powerful tool in the hands of a capable systems architect, who actually needs it. For the rest, it's pretty much like an adult male tiger; excellent to watch in its natural habitat from a safe distance, terrible pet idea for you and your fiancee (and not because you live in a studio apartment.)
1) It's a networking library
Yes, but one that's easy to use. Versus BSD sockets, which have an annoying array of quirks when programming for multiple platforms.
Moreover, it's consistent whatever your transport. One thing I've found using zeromq is that, because it's so simple to use, it makes you want to think in terms of message passing. I use a queue to push messages between threads at practically no cost. Then later if I feel like it I can shunt the consumer to a different process, or a different host, or several different hosts with a load balancer. I could do the same with raw sockets, but I'd rather eat my own leg with a spoon.
2) It uses a binary protocol. Good luck debugging that with syslog.
It's just a packet. You can put whatever you want in it: XML, JSON, protocol buffers, msgpack, JPEG.... The wire protocol is trivial.
It also has jolly nice bindings for Python, Ruby, Lua etc, allowing you to bridge between C or C++ and your scripted components with zero hassle.
Go on, give it a go.
STOMP! Stomp is simpler than 0mq, but you have a central server instead of point to point.
and STOMP:
ZeroMQ: Send a message, other side either receives full message as a single payload (or nothing at all? or an error is raised? I'm not tototally sure).
Sockets: write(), read() and friends. Other side has to make sure he has the whole message, otherwise read() again and wait, deal with errors on its own, etc.
It's right there in their <title/>: "Fastest. Messaging. Ever".
The great majority of the people would just benefit from learning socket programming better, and sticking to something native to their tongue. Use the messaging infrastructure that integrates best with your development process, or opt for something with more padding, like Rabbit.
Call me weak, but I like infrastructure software whose guts I can peek, lsof, cat, grep, log, strace and diff. Stuff that I can take a snapshot of and reproduce. Network services should be transparent to the application code base. I wanna poke a whole in my firewall, give some hint to inetd, a few lines of cron and be done with it. I simply don't trust my own software enough to be sure I will not screw up the infrastructure.
That said I am a little envious when I read that I can send("bunch of shit") and receive() and call it a day. I wouldn't call that premature optimization, nothing is going to beat raw sockets and domain specific knowledge.
That said, since you brought up AMQP (we use Rabbit, too, I'm a fan) have you seen this?: http://lists.zeromq.org/pipermail/openamq-dev/2010-March/001...
Also, I have already stated that it was a powerful tool in the hands of a "capable systems architect". Besides, I am a closet iMatix fan ;-)
http://news.ycombinator.com/item?id=1230509
All I wanted to point out is that ZeroMQ should not be treated as another "plugin" that people are encouraged to use willy nilly. I would use it in high-performance applications in financial trading, VOIP, scalable game servers, etc.
But these are not what most people are working on. Blog posts like these just encourage the ADHD bleeding-edge types to take another stab at their own thighs in the unjustified pursuit of "web scale". And the moment SmashingMag gets a hold of Zero, we will see the most gruesome newbie bloodshed since the battle of WordPress.
As a programmer, using 0MQ is Better than Sex. You can quote me on that. It would be unethical to deny such fun to anyone. What beginners need, with any new technology, is a set of clear recipes that they can implement without having to invent everything. This was why a books like Stevens' trilogy were so vital to getting TCP out of labs and into the grubby hands of ADHD bleeding-edge types.
We're working on that, with books like http://www.zeromq.org/docs:user-guide. That'll grow to cover all of 0MQ eventually.
Would you need to use a broker for that ?
And no, you don't need a "broker" for that. The messaging middleware application of ZeroMQ, and other middleware, is not the same "messaging", in the human communication sense. The former can be used to implement the later, but typically it's for inter-application messaging and communication.
In your case, I recommend you use a tradition RDBMs and do something like
CREATE TABLE Message
(
messageId SERIAL,
senderId VARCHAR(20) NOT NULL,
recipientId VARCHAR(20) NOT NULL,
timestamp TIMESTAMP DEFAULT NOW(),
message TEXT,
status message_status, //
PRIMARY KEY (messageId)
);
CREATE SEQUENCE serial;
CREATE TYPE message_status AS ENUM ('draft', 'sent', 'new', 'read', 'archived', 'deleted');
If you have a persistent connection, then the code that inserts to the database also puts out a ticket in some "refresh-display queue" in the following manner. All logged in users are eligible for message status updates. i.e. those that have an active session will get "you have new email" message. You will have an alerter process/thread/job that runs in the background and reads those message update tickets. It takes the recipient ID and updates that user's display, i.e. pushes.
Typically, you would use ZeroMQ in just that later part; as a queue of updated-message statuses, and the queue is consumed by a background job that just notifies a bunch of remote users that they have new updates. Javascript on the client side takes care of the display update/modification (typically a DOM operation, though you could also have a flash or java applet consume pushed data; those later two can actually keep persistent connections, but watch out for session length or you might quickly run into file descriptor lossage.)
I'm just throwing out something random here, but I imagine the reason no one else really does this in standard libraries is because it's quite complicated and can easily lead to fragile protocols. Take AMQP, for example, it took something like 5 years to agree on their 1.0 standard.
What JMS achieved was to somehow turn these two very different semantics into a single one, based around "destinations". That was very clever but also makes JMS weirdly complex to use because the queue (request-reply) pattern and the topic (publish-subscribe) pattern just don't work the same way. We designed AMQP originally by taking the JMS spec and reverse engineering that into a wire level protocol. Then around version 0.3 we threw out destinations and came up with a generic wiring model based on exchanges, bindings, queues. That was my summer holiday in 2005.
AMQP is BTW still some way away from a 1.0 standard and mainly it's been five years trying to get reliability working in the center of the network. That always seemed destined to failure, as I explained in a presentation to the working group in 2006: http://www.zeromq.org/whitepapers:switch-or-broker.
AMQP and JMS both focus on resources held in the center of the network. It's familiar to anyone using HTTP as their stack. Thin stupid clients talking to big smart servers. 0MQ turns the focus to smart clients working across a thin, stupid (but massive) network.
Both approaches have their value. We (iMatix and the 0MQ community) tend to believe we can do a lot more, faster and more cheaply, using the distributed approach.
I've worked with JMS, and no. Maybe my implementation was flawed. (Weblogic) IIRC, you needed 5 different objects of 5 different classes just to do anything.
Perhaps I'm missing something, but wouldn't this make ZeroMQ a Modern & Fast abstraction of a Networking Stack?