Hacker News Reader

Comment by CharlieDigital | Hacker News Reader

Back in 2014, I was rebuilding the core of an event processing engine. At the time, the decision was between Apache Kafka and rolling our own. After investigating Zookeeper, we decided to roll our own and chose ZeroMQ as the messaging layer as our on-prem customers probably didn't want to own and manage Zookeeper.

ZeroMQ was absolutely solid and stable; incredibly trouble free and the only issues we ran into were when IT teams didn't open the ports we documented in the configuration procedure. (The resultant architecture actually looks a lot like Flink)

In any case, ZeroMQ is a fantastic piece of technology that I feel like I don't see out in the wild quite enough. It's dead simple and incredibly stable from my experience.

View on HN

The problem is that it started out as any but stable and reliable. It asserted on received data, which in a network application is a super newbie mistake. When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection. So the zeromq guide which said "look how easy it is" was only true if you ignored errors. If you are writing network code and ignore errors, well, good luck. That was a long time ago (~10yrs) so if it is better now, good for them. Also, both founders have left the project. One passed from cancer, the other didn't like what he built and started over in C. Not that they can't be replaced, but transitions can be hard and take time.

> When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection.

You just gave me openstack+zeromq flashbacks

Yes - nanomsg / nng are quite good. I think / believe they'll be part and parcel for some space missions soon enough.

> It asserted on received data, which in a network application is a super newbie mistake

What do mature network applications do instead?

What I mean is literally have an assert with incoming data as the parameter:

> assert(data_buf[4] < 8);

While your protocol might guarantee that data_buf[4] should always be a value less than 8, you don't use assert() to check it because it aborts the program if the check fails. The proper thing to do is a range check that returns an error for a protocol error (malformed data, etc.).

ZeroMQ literally called assert and any bad data coming in over the wire would cause your app to abort. Insane.

Here is an example bug report:

https://github.com/zeromq/libzmq/issues/186

Keep in mind this was a LONG time ago! So this is not an indictment of the current project!

TCP asserts on received data, so I guess GP really meant reliable message queues. They generally assert after commit to disk.

What does it mean to assert in this context? I assume not the same thing as C's assert macro; is it the EE sense of "pulling a pin high / low"?

I literally meant the library would call assert() on incoming data. I am fairly certain that has been removed for a long time, but it can be hard to get past first impressions.

I misunderstood assert as part of the protocol, i.e. the latter.

In a company I work for they decided to do the same, around the same time. I believe it was a wrong call. Over time requirements have grown and we ended up bolting all kinds of kafka features on top of the zeromq thing, but of course much crappier. And in the meantime kafka doesn't require zookeeper anymore and is the de-facto standard

Of course, ZeroMQ and Kafka are two very different tools that serve different purposes and one needs to understand the tradeoffs.

For us, delivering an on-prem commercial off the shelf solution, it was untenable to expect the customer IT team to operate a separate, relatively huge piece of tech (remember, this is 2014). Maybe the heuristics would be different today with K8s and advancement of Kafka. But ZeroMQ as an in-process, distributed messaging layer is dead simple. If your use case requires anything else on top of that, it's on the team to design the right solution like resiliency, statefulness, etc.

For a high throughput, distributed compute focused use case, I think ZMQ is still a great choice for coordinating execution. But Kafka and other options on the market now are great choices for higher order abstractions.

I worked on a similar greenfield project around the same time and we looked at RabbitMQ and Kafka, eventually going the RabbitMQ route. We were also developing an on-prem COTS product, and zookeeper played a big role in our decision to go with RMQ, not to mention at the time CloudAMQP had a very generous free tier (not so much anymore, but it's still okay-bordering-decent). No one install would ever hit the scale where Kafka makes sense so I still think it was a good call pushing 10 years later.

A company I worked for had the same problem. Messages were being dropped, and either no one on backend knew how or wanted to investigate. I was on the data team and we just had to deal with it.

It's also quite fast.

I remember we had a similar use-case. This was for collecting live statistics from a sizable Varnish Cache cluster. We wrote a in memory database to store the data. It's been chugging away for about 10 years now and last I heard the zmq traffic alone was about 3Gbps with zero issues.

Aside from some scaling issues, it still is a great solution for having real-time insight into the performance of Varnish and the cache.

We even went as far as writing a Golang version of the VCS server to better handle some of the scaling challenges. I don’t remember the exact library that we are using to call into ZeroMQ, but the CGO overhead was minimal.

> It's also quite fast.

IIRC it does all I/O on separate background threads which means every operation actually goes through an inter-thread queue. Which can be good and bad.

If you look close at the neuralink demo you’ll notice they use zeromq. I suspect the flexibility and the reduced latency makes a difference.

The problem with ZeroMQ is that it is highly opinionated about threading and concurrency in a way that doesn't necessarily mesh well with other components of your application stack that have their own opinions

http://wiki.zeromq.org/blog:multithreading-magic

There's nothing wrong with the opinions presented there -- but it really wants to own your whole threading / serving stack, and isn't really compatible with e.g. a Rust async tokio binary where threads can "move around" etc.

Well, the original non-thread-safe sockets (e.g., ZMQ_SUB, etc.) were that way because they support multi-part messages. And yes, the fact that they are not thread-safe is a bit of a stumbling-block for ZeroMQ newbies (including me, back in the day).

However, it's not that hard to work around their non-thread-safe nature with a single dispatch thread (e.g., see <https://github.com/nyfix/OZ/blob/master/src/transport.c#L457>), and that approach has other potential benefits.

In any case, newer socket types (e.g., ZMQ_CLIENT) have since been defined that are thread-safe, but necessarily don't support multi-part messages. (They tend to be different in other ways as well -- e.g., ZMQ_RADIO/ZMQ_DISH are "sort of" replacements for the original ZMQ_PUB/ZMQ_SUB sockets, but have other constraints as well).

At my day job we run > 50 million messages per day through ZeroMQ, and have been quite pleased with its performance and stability (https://github.com/nyfix/OZ/tree/master#who-uses-it).

But the truth is that ZeroMQ has a steep learning curve, and some rather sharp edges that can leave nasty cuts if one is not careful.

If you're interested in becoming familiar with ZeroMQ, you might want to check out our implementation: <https://github.com/nyfix/OZ/>.

If you have played around with GNU Radio, ZeroMQ is baked-in for communications with outside applications. I've played with it a bit, and found it to be very much fire-and-forget once you have it set up properly.

That's cool, I've played with GNU Radio some, wasn't aware it used ZeroMQ, actually, I've used several *MQ's but not ZeroMQ, guess I should look into it more.

I wonder if this is one of those problems Erlang and the Beam VM solve out of the box. Native clustering, message passing from one node to the other, no need for a separate dependency... That's the same design, isn't it?

Disclaimer: I don't work with any of that tech right now, but I'm looking forward to working with Elixir, which is in the same ecosystem.

NATS is today's ZeroMQ, even easier and more reliable.

Is it? ZeroMQ runs in-process as a library. NATS looks like it requires a server.

Fair.

Having used both (from Python) I found NATS much better. ZeroMQ in particular caused a lot of problems around the HWM (High Water Mark) limit for me or around process starting order (who creates what channel). Having a separate server helps with the last problem.

ZMQ for me fills a very specific use case of needing high-throughput, _in-process_ distributed messaging.

I think once the _in-process_ constraint is lifted -- you can install or rely on an external messaging server -- then the field becomes much wider in terms of solutions to pick from.

BTW, the way we solved for a similar HWM issue is that we decoupled the ingestion of events from the distribution of said events (with ZMQ). So one process was ingesting events and would send it to a coordinator process that would then send events to processing nodes. The coordinator would reply with it's current queue size and the ingest process would do a back-off once it reached a threshold. This allowed us to stop/start any node in the cluster in any order.

No. It doesn't.

Sure, to get the most out of NATS you should run it as a server (which frankly isn't difficult as its a single Go binary).

But you can embed it if you so wish. Indeed if you look at the NATS source code[1], that is exactly how NATS do their testing. They use an embedded NATS server in their test routines.

[1]https://github.com/nats-io/nats-server/tree/main/test

Exactly, zmq is far more flexible in deployment options because of this.

We (at work) use ZMQ + our own special sauces to run vehicle automation messaging stuff. Running some enterprise-ish broker in that environment just isn't on the menu.

Reply on news.ycombinator.com