NSQ – A realtime distributed messaging platform

This is something that I've always tried to stress when talking about NSQ...

The "message queue" is the most boring and uninteresting aspect of the system.

It's the combination of out-of-the-box tooling and conceptually simple primitives that really differentiate it from other systems.

https://github.com/simplereach/chef-nsq https://galaxy.ansible.com/list#/roles/2265

I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1].

While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful.

These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most.

P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance.

[1] He's since posted this follow-up that outlines many of the same things http://www.bravenewgeek.com/benchmark-responsibly/

I'm also curious if you've had issues with duplicated delivery, where a message is delivered more than once, and how you've handled that case.

NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1].

Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools).

When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees.

[1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.

I don't know about NSQ — it looks nice, but the lack of true persistence and ordering means it's not something I can use. For example, lack of ordering means you can't use it as an event bus for document-oriented changes.

I know it's not a popular sentiment these days, but I prefer to err on the side of correctness. I have had enough of systems that are supposedly SPOF-free but still fail for mysterious reasons and are nigh-impossible to debug once they do (ElasticSearch, ugh).

RabbitMQ's lack of transparency means it's harder to work with the contents of a queue. For example, if the queue processing has stalled for some mysterious reason, what's clogging it? With RabbitMQ, the only way to peek into a queue is to actually pop messages from it (using a tool such as rabbitmqadmin or amqp-dequeue).

Another problem with RabbitMQ is that messages go away after being acknowledged. I, for one, would like to see a bit of history. Sure, you can build this into individual apps, but it turns out this kind of history is beneficial for many kinds of apps. There's DLX, but you can't (afaik) set up rules to automatically copy acked messages into a history queue.

I can't say I like AMQP. Binary protocol, requires several layers of client tooling to work with. If anything goes weird, strace or tcpdump are of no use. It's also a very complex protocol whose interpretation and matrix of support features seems to change a lot.

I'm currently writing a small job manager in Go that's backed by PostgreSQL (at least initially) because I want something safer, more stable and more transparent. It's based on my realization that jobs are different from events, and should be treated differently.

For example, RabbitMQ has no conception of scheduling or prioritization, and messages are ephemeral objects that go away once they've been acknowledged, as opposed to jobs, which have a life before, during and after processing.

So far, it looks pretty good. Postgres can't process a bazillion messages per second, and it's not going to work for "big data", but it seems to scale decently enough.

However while my queue project supports persistence, synchronous replication, delayed jobs, at least once and at most once delivery semantics, automatic federation, there is one thing I don't support in the list of features you mentioned: ordering. And I believe there is a strong argument for not supporting ordering in certain kinds of message queues.

By implementing only a weak form of ordering (approximated wall-clock ordering, so that, usually jobs are served in roughly insertion order) we gain: availability (as in CAP, the system can continue with a single node), latency (even in the case of synchronous replication, you need to care just for a message to be replicated into N nodes, regardless of what those N nodes are), and functionality (the queue can auto-reissue messages after a specified retry-time, so at-least-once delivery is trivial to accomplish for consumers and producers). Depending on how the system is designed, to give up strict ordering also wins you scalability.

So I agree about your reasoning but my feeling is that message ordering is a big point that really changes how a message queue is shaped. My bet is that there are many problems where ordering is not needed but all the other features are.

Somewhat on a tangent, one of the goals of the design was to avoid any servers in the middle of the "workers" (consumers) and producers, and making it such that any consuming or producing server dieing or experiencing a network partition doesn't slow down any producers or consumers who can still reach each other.

You're right, this isn't really focused on replacing map-reduce thing like some other message streaming systems are. For one, I can't imagine a "sorting" step using nsq. It's still used for somewhat hefty data-processing tasks, in multiple stages. Each stage has as many servers spun up as needed, consuming messages directly from the previous stage, and publishing to nsqd on localhost. They might consume and acknowledge multiple messages to produce one new message.

For reduced inter-server hops before hitting the database, you can run workers on the same host as producers. So you might have a X hosts producing messages (and running their own nsqd locally, of course), and have Y consumers on each host, configured to consume messages from localhost instead of using nsqlookupd to find all sources, and those consumers would make direct requests to the appropriate database host. With this setup you can still use nsqadmin to monitor the queue levels on all producer hosts, and you can still be archiving messages to disk on some other remote hosts.

For different applications differently delivery QoSes are tolerable. For instance, if your messages are idempotent (either by design or nature) then "at least once" is equivalent to "exactly once" and may be much faster/easier.

Generally, "exactly once" demands the least of the application and the most of the broker.

[0] I suppose you could have "any number of times" as well, but that's no guarantee at all.

However, it is a benefit in that allowing for more-than-once-delivery affords better performance. Think of it from an algorithmic perspective: if you want to send a message exactly once, you have to wait around for acknowledgement that your message was accepted.

If you instead don't mind sending duplicates, you can fire off the first copy, wait for an ack for a while, and then fire off again to another node (in case, perhaps, there was a network partition). If it turns out that both copies arrive at their final destination intact, your application now has to deal with duplicates. But, that's faster than the alternative.

So it puts more burden on the developer to deduplicate where necessary (or program in a model where duplicates don't matter anyway, such as with CRDTS), in exchange for greater throughput.

The network could cut at any moment during a transaction.

If message consumers don't send acknowledgements, and the queue considers the message sent as soon as it has been successfully written to the network connection, then it might never really be delivered and processed. You can't be sure if it really got to the destination before the network cut out, or something crashed. This is "at most once" - it may have been processed once, or not at all.

If message consumers acknowledge messages after they've finished processing them, and the queue waits for that, then the acknowledgement might be the thing that is lost. The queue sends the message to a different consumer if it doesn't get an acknowledgement before a timeout, but the message might have been processed already, and the consumer crashed or the network partitioned just before the acknowledgement. This is "at least once".

You could consider a final scheme - consumers acknowledge receiving the message, before they process it. The process could crash just after the acknowledgement, or the ack could be lost without it knowing. So this isn't "at least once" or "at most once", it's "no idea".

Or maybe they even wait for an ack that the server got the ack, before processing. The process could still crash after sending its ack or after receiving the other ack, this is still just "at most once".

"At most once" is usually not useful. Usually you want "at least once", and then logic specific to your data to resolve "more than once" in an acceptable way.

https://news.ycombinator.com/item?id=4631994

Although I don't know how the system has evolved since then.