I've been doing a lot of reading on it, and am now curious if anyone has used it and encountered issues they could share?
Seriously, 0MQ has made network programming fun (again) in a bad, addictive, way. Any design I can think of turns into real working code in a few hours, sometimes days. And I'm using C, a language that isn't normally fun to work in.
Right now, it's multithreaded clients and servers for resilient shared distributed hash maps. Tomorrow, network-wide logging. After that, another message broker. And so on.
Yes, it's a negative experience. I'd like my old lazy life back.
For the love of god, don't try it.
This post on SO outlines some of the troubles I've had: http://stackoverflow.com/questions/4870814/is-zeromq-product...
A couple of other issues. First, it's really easy to game a ZeroMQ socket. Send it the wrong data, and your application will just fall flat on its face. So you can't run it anywhere that has untrusted computers (e.g. over the Internet).
Another issue I've had is a race condition that occurs when you call recv() before anything is in the queue. The method will continue to block even after it receives something. This is a big deal because it requires some workarounds with bad performance. But I wasn't able to get the jzmq dev team to reproduce it, so it must be something restricted to either OS X or just my system.
FWIW, I think most of the issues are restricted to jzmq, because there's a good deal more complexity running around in that project to overcome the Java <-> C bridge.
The reason I continue to use it anyway is because:
1) It's absurdly useful when it works.
2) The dev team is very responsive.
3) Bugs do get fixed if they can reproduce it. I already have had one issue resolved: https://github.com/zeromq/jzmq/issues/closed#issue/31
As for Tornado, I am in love with that technology. Rather than a framework, it's more like a set of libraries for HTTP communication. That has huge implications, and it feels much more pleasant for me to work with than, say, Django.
Everything I'm doing is open source, I'm planning on using the Apache 2.0 license. The Github repo (without any code yet) is here - https://github.com/joerussbowman/Scale0
The README gives an overview of what I'm attempting to accomplish.
1. Extensive use of asserts in release builds terrifies me. It's meant to check for conditions that shouldn't happen, but I see users complaining about their apps aborting with assertion failures on ZMQ mailing lists and it comes up fairly frequently in Google. There are a fair bit of asserts for error codes returned from system calls. I don't want a critical process crash because I've used library in a wrong way in a completely different part of the application.
2. Only in 2.1 they've fixed the problem where some messages would not be flushed and be lost if you terminate the process too early. This seems like a fairly common bug for younger projects. Recommended workaround is... calling "sleep" before you exit, which is one of the deadly sins of multithreaded programming! This and above point convinces me that ZMQ isn't as mature enough for me to be comfortable with.
3. Transparent reconnection is good, but some of our applications need to quickly detect that other nodes in the system are missing, which forces me to implement off-band heartbeat mechanism.
4. Threading model seems a bit awkward to me (last I checked). First of all, let me state that I personally believe that a library starting threads behind your back is a Bad Thing (unless it's a framework). ZMQ uses a sender thread that you queue your messages into, yet it forces you to dispatch your receive loop by either blocking read or zmq_poll. If it already starts threads by itself, why not provide a callback?
5. Not really a problem, but a missing feature: no way to demultiplex messages from a stream of messages, so you have to implement it yourself. You can subscribe to a subset of messages on a socket, but can't subscribe to multiple subsets from a single socket.
I use tons of debug asserts in my code, but I'm slowly coming to the conclusion that asserts enable lazy coding. Most asserts should probably be replaced with proper logging and error codes.
1. The 0MQ devs originally got asserts backwards, using them to validate external input (e.g. on sockets) instead of internal consistency. We've been fixing this for a year or two now, and it's pretty good. You'll get assertion failures if you e.g. use sockets from multiple threads. Not so much if you pass bad stuff onto sockets.
2. 2.1 was a great step forwards, and the use of "sleep" was in toy examples. Real networking apps tend to run forever, so this message loss at exit wasn't a big deal. You're right that the product is still young.
3. Totally agreed, this lack of peer presence detection is annoying, and the source of some debate on the lists.
4. Threading model works fine for me, I've used it extensively. A usable reactor is a hundred lines of code, no more. See the libzapi zloop reactor, in C, for example.
5. Demultiplexing sounds like useful functionality but should probable sit above sockets.
We evaluated using ZeroMQ for inter-shard communication in our distributed database engine, and while ZeroMQ had a lot going for it, in the end everyone on the team was so frustrated with attempting to reproduce corner case assertions and debugging ZeroMQ internals that we decided to bid it good riddance.
We were working towards a deadline, and ZeroMQ in the end proved counterproductive towards that goal; during the time we used it, we ran into some half a dozen different ZeroMQ asserts, some of them very difficult to reproduce, and of which I believe we managed to solve or work around only two or three.
Some of the problem may be in expectations. If you go in expecting it to be a mature and solid black-box solution (as its versioning might seem to indicate), you may be expecting too much. If you mentally prefix a '0.' to the version number and think of it as fast-evolving alpha software, you'll be happier.
I still believe the concepts behind ZeroMQ to be viable and laudable ones. We may reevaluate it in the future, but not anytime soon.
It sounds like you should investigate http://mongrel2.org/home
Right now thougt I'm thinking learning experience with zeromq, and also I'm seeing how I can build something that can be used to scale an http application beyond a single datacenter/cloud and more. I actually find that pretty exciting and since I have young children and a good paying job, I'm still treating the product I'm working on as a hobby rather than a business.
And who knows, maybe you'll do something better. That's progress.
Also check out gevent and eventlet to see the differences with those systems as well. I just submitted a patch to eventlet to give it better zeromq support.
Thing is 0MQ gives you transparent auto-reconnection--but I want to indicate with a spinner when that's happening--and it makes request-reply synchronous--but I already do everything asynchronously in the client anyway. Hmm.