ØMQ – The Guide (2011)
zguide.zeromq.org
zguide.zeromq.org
I believe the original developer passed away a few years ago?
EDIT: This is the post I was looking for: http://hintjens.com/blog:115
Yeah, via euthanasia.
nkurz on Apr 18, 2016 [–]
Thanks. Does anything come to mind that you should have done more of (or instead)?
PieterH on Apr 18, 2016 [–]
Probably more frequent checkups, to be honest. Though it's probably moot. "
Martin Sustrik was actually what could be referred to as the lead engineer for the initial ZeroMQ project, though he left it quite some time ago.
Zeromq is a bag of tools and it's not really clear to me how a novice would put them together safely. This is important for a message system because everyone has to start somewhere and it really shouldn't be "step 1: build your broker with our subroutines". Is that broker really going to be any good?
Secondly, for many years zeromq was a kind of monty python style 'judean people's front vs people's front of judea' community. They split from the AMQP standardisation process I agree had clearly gone wildly wrong and then...split and then...split again.
Right as far as I know people doing messaging are using a) RabbitMQ if they're deploying it themselves b) AWS of GCP's thing or c) some proprietary thing which has no name recognition among the public (who here has heard of AMPS?) d) Kafka. No one is using zeromq as far as I know! Look forward to anyone sharing their positive experiences using zeromq on big projects eg in banks.
"Bag of tools" is about right. Where RabbitMQ/AMQP or MQTT or DDS can, depending on exactly what I want to do, provide me an end-to-end solution where, with a bit of configuration and network management, throw in a message and expect it to go where I want it to, ZeroMQ hands me a pile of pieces with which I can, presumably, build a thing that will do the same thing. That's not something I've ever found that I need-- either one of the premade solutions fills my needs, or I go and write something myself and skip ZeroMQ.
I'd happily use it as the glue between two processes that I control, but would grab one of the other tech you mentioned for anything exposed on the boundaries of my app.
We’ve used ZeroMQ in production as an alternative to HTTP message passing and it was great for that. We would build something like RabbitMQ atop of ZMQ, the only similarity between the two being the name, really.
So it is an alternative to (eg) Google pubsub and I think can be compared with it. Is there any case where it would be used for a different /purpose/ than the others?
As in, in building your RabbitMQ competitor HareMQ, you might build on top of HareMQ. You can't just use HareMQ because you haven't built it yet.
Maybe there is no guarantee, but it can handle unreliable connection just fine, at least when I used it a few years ago.
Pseudocode:
Send msg to server
if socket timeout:
reinit socket and send again
Recv from server
if recv.msg != 'ok':
send again
else
remove msg from storage
These may all be fine for your application, and then 0mq is an excellent fit. But I think that for most applications, having some kind of pre-implemebted solution to all of these problems, even with slightly more overhead, is extremely valuable.
True, better send twice than never.
> the client connects later than the server send
Server only responds, never sends anything.
> overwhelming the network
"broker" is handling it nicely
I strongly agree that zmq will not fit in every use case. Always search for best option.
> How do I determine how many messages are in queue?
> This isn't possible. [...] rather than provide incorrect information the library avoids providing any view into this data.
> How can I retrieve a list of all connected peers?
> This is not supported.
Those kinds of decisions are very scary for production environment. We've had to debug network connectivity problems before, and it just possible what all the interfaces Linux kernel provides. I cannot imagine doing it with library which hides stuff from you on purpose.
Setting up a monitor socket allows you to observe pretty much any event you could care about, like peer connect/disconnect/errors, etc.
There is also a draft feature to access the number of queued incoming/outgoing messages (see https://github.com/zeromq/libzmq/blob/master/doc/zmq_socket_... for details).
> At any given time a message may be in the ZeroMQ sender queue, the sender's kernel buffer, on the wire, in the receiver's kernel buffer or in the receiver's ZeroMQ receiver queue. Furthermore, a ZeroMQ socket can bind and/or connect to many peers. Each peer may have different performance characteristics and therefore a different queue depth. Any "queue depth" number is almost certainly wrong
The library doesn't hide stuff from you, it just refuses to show you information that might as well be a random number.
If you want to know how many peers are connected you use heartbeats instead of just blindly trusting that 1 tcp socket = 1 available peer.
Edit: just checked the FAQ and apparently it will now block or drop messages. I sort of wish it would explicitly error instead of block, so as not to require the caller to queue in a thread or something.
How do you know my circumstances? Maybe this system has tuned rmem_max/wmem_max so I don't care about kernel buffers? Maybe my programs have problems with multi-gigabyte queues? Why are you assuming stuff for me?
The right answer of course is you don't expose a single "queue size" value, but you don't hide it either. Instead, expose something like "list of sockets: for each, here is an fd and a size of userspace queue". I'll then query the kernel myself to get sender's and receiver's kernel buffer size and so on.
Life is hard enough without libraries hiding stuff from you!
Other than that, great piece of software.
[0] - https://lucumr.pocoo.org/2012/6/26/disconnects-are-good-for-...
Interesting that nanomsg tries to better zeromq (https://nanomsg.org/documentation-zeromq.html) and then nng tries to better nanomsg (https://nng.nanomsg.org/RATIONALE.html)
edit: just noticed nng and nanomsg share the same domain, so probably the same people behind them.
Edit: Found it. https://250bpm.com/blog:4/
I've seen this sentiment before from C enthusiasts. Maybe it makes sense on the Linux kernel, where you don't want to keep telling contributors not to use the most common C++ features.
But I don't think it makes sense on very small projects, and maybe I just draw the dividing line for where it stops working at a different size of project than he did.
[1] Exceptions suck and constructors suck and C++ would be better if it was just Rust with no borrow-checker.
My takeaway is that even though these newer libs seem to be simpler and avoid some issues, they've just never caught on.
zeromq is the same way. It's not a replacement for something like kafka or rabbitmq, it's a replacement for socket()
”ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker.”
I read it as a low-level abstraction for when you want to build a custom messaging infrastructure.
I prefer the “replacement for socket” pitch, personally.
It does bring some nice things to the table, but using it has always felt at least a little bit "weird" to me. The docs are a little weird, and there are random caveats sprinkled around the library (e.g. you don't need to synchronize the startup of dependent services like described above, unless you were using certain transports in older versions and so on).
It used to be marketed somewhat towards developing multi-threaded applications, but it was always rather unclear what exactly the advantage of using, for example, the comparatively slow PUSH/PULL socket (ipc:// are pipes) over a MPMC (or as needed) queue ought to be.
It is also marketed towards resilient distributed apps, but, perhaps I was holding it wrong, but it'd always behave much like TCP and stuff like REQ/REP would pretty much always block forever if the peer goes away, while stuff like PUSH/PULL isn't reliable when peers go away, so for a distributed app these can't really be used anyway. So for this use case it always seemed rather too low-level and its guarantees too weak / non-applicable.
I had exactly the same (disappointing) experience.
unless you tell it to do otherwise, that's exactly what it will do. The guide covers this: 4. Reliable Request-Reply Patterns
zmq appears to be a layer that tries to hide some of the most important details that the socket layer exposes. socket programming and the state machines that tend to go along with it are some of the least difficult parts of systems programming to do well, it is well documented, well understood and there are ample tools to help measure ground truth. it was always weird to me that zmq had such a strong reputation; it basically hides what would otherwise be a few hundred lines of library code (including a framer).
Like you can tell it to use sockets if the machines are remote, or different inter-process communication if they're running on the same machine, etc.
It also has some common patterns like pub/sub and others.
Some places you might use it: real-time financial data streams, multi-agent worker task/management, distributed database syncing. It’s basically a networking lib that makes common patterns easier. E.g send this message to everyone, list connected peers, handle these messages this way if the peer can’t take the message, and so on. They support doing this in a bunch of languages over a bunch of communication protocols.
Now all the Async/coroutine/ipc verbiage is mostly just because synchronization is a communication problem which is what zeroMQ happens to solve as well.
It hides the scalability problem of having many TCP sockets without solving it (e.g. pub-sub over TCP).
It does not solve the ahead-of-line blocking issue.
It requires reading a lot of documentation and doing experiments to find out what is the right combination of " fan-out, pub-sub, task distribution, and request-reply" that you have to use.
e.g. https://zguide.zeromq.org/docs/chapter4/ https://zguide.zeromq.org/docs/chapter5/ https://zguide.zeromq.org/docs/chapter3/
Plus a lot of "magic" happening in the library involves queues that you have no control over. You get applications that hang and fill buffers instead of failing early.
I suspect this is the reason why it never became a new layer in the TCP/IP stack
ZeroMQ is not an out-of-the-box MQ solution. It takes some work to understand and build a functional MQ solution with ZeroMQ. If you want something that works out of the box I recommend RabbitMQ.
Are there are any, preferably open source, message queue based messaging systems using ZeroMQ?
It also allows one machine to broadcast a message to a 'topic' that other machines listen for.
Many other things. It's not perfect, but it Just Works.
A lot of people these days might use something like flink or spark to build a pipeline whereas I'd prefer to write a number of small servers with various fan out or fan in operations, broadcast style operations, request/reply semantics etc where appropriate
Makes way more sense and much simpler to my mind and allows much better control over performance characteristics.
1. A lot of gotchas and surprising behavior. For example this:
> To be more explicit: after calling zmq_send the socket may become readable (and vice versa) without triggering a read event on the file descriptor.
2. The library will assert internally without actually giving you any clues what did you do wrong as a user.
I would rather spend some extra time in development by using pure sockets rather than the stress of debugging problems, arising from the above, later.
I wasn't a part of the initial decision making but was pretty happy with how easy it was to work with both zmq and flatbuffers.
It’s not an unused letter that you can snap up as a cool way to write zero. People will read it, some will recognize it and assume it has meaning, because it has. That’s how letters work.
There are zeroes with dashes through them, but if you look closely they almost always take care to keep the dash inside the (non-)circle zero to make the difference clearer.
I realize most people in the world won’t care about this, but with Northern European eyes, this looks stupid.
> The Ø in ZeroMQ is all about tradeoffs. On the one hand this strange name lowers ZeroMQ’s visibility on Google and Twitter. On the other hand it annoys the heck out of some Danish folk who write us things like “ØMG røtfl”, and “Ø is not a funny looking zero!” and “Rødgrød med fløde!”, which is apparently an insult that means “may your neighbours be the direct descendants of Grendel!” Seems like a fair trade.
Edit: missed an s in dessert.
That is the meaning everyone that does not primarily use the Northern European alphabet (aka most people in the world) assume it has.
Maybe suppress your instinctive outrage next time, or if you really need to be angry at something, at least direct it at the right place: the dead mathematician who proposed it [0]
Maybe suppress your instinctive correction next time until you know that the person you’re correcting is in fact mistaken. Or just use a nicer tone to begin with.
I don't know what is more funny ("haha") - people not being able to pronounce it or how stupid hard Danish is to speak.
we have https://xkcd.com/927/ with sockets in frame 1, and zeromq in frame 3
Are there some clear wins of zeromq over sockets/udp/tcp?
Hmm.
The wide range of language support was also nice. It allowed me to test the interface using a Python notebook which came in handy for debugging.
For me 0mq works internally, and not for interfacing outside a closed system. I think that might be the source of some confusion
What are the delivery guarantees for basic http?
The 1st issue is that ZMQ implementations generally have an IO thread handling all socket IO. REQ/REP patterns tend to work out just fine, because the client is going to wait for a reply from the server. But things like PUSH/PULL become...harder to follow. You can push things in the client, but how do you know when they've flushed out of the IO thread and made it to the server? Those sorts of things matter when shutting down a process, running integration tests, etc. In contrast, with an HTTP request, you're basically always doing REQ/REP, and so you know when the data has been pushed.
The 2nd issue is that ZMQ implementations tend to be harder to observe / operate than a more common path like HTTP requests. (HTTP requests can go through a load balancer, have standard response codes, use headers for authorization, etc.) For these reasons, I've tended to avoid ZMQ ever after -- it seems like often, you're best off either using REST or GRPC if by HTTP, or raw TCP if it's purely a data push kind of operation (e.g. forwarding structured logs, with framing, to a remote TLS endpoint).
Surely that implies incorrect use of the socket types vis a vis their design characteristics?
- I wanted easy message framing - I didn't know much about networks - HTTP servers that can be embedded in C++ aren't great, and I didn't know anything about HTTP - It is nice that you can ignore application startup order if you don't care much about reliability, which I didn't
If I knew then what I know now, I would probably have picked something else. For one, I couldn't make it integrate with my main event loop, so I had it block in another thread, and that caused a problem of inter-thread communication that was nearly as bad as the original problem, for me as an inexperienced noobie.
You can get a file descriptor from ZMQ to use with your own event loop (depending on your event loop of course). I have done this successfully in the past with an epoll (iirc) based event loop.
The documentation is at [1], the ZMQ_FD option. You do have to read the instructions very carefully though.