Hacker News Reader

If you were capable of replacing your use of rabbitmq with 3 lines of python and a redis instance, you shouldn't have been using rabbitmq to start with.

That doesn't mean redis is a drop in replacement for any of the valid uses for rabbitmq though...

can you share some of those feature where rabbitmq that is useful? Tks.

Sounds to me like the problem is, that the company taking out the cannon to shoot at sparrows and that nobody seem to have bothered to provide working dev environments?

It feels common that people think they their every problem requires a planetary scale solution or one that handles every conceivable case that could occur before the heat death of the universe. Be that because your a startup and think you are going to need to support a hundred million concurrent users on launch day, or because your an enterprise and think because your are oh so important you need to use the same tools other important companies use.

My own anecdote: When we refactored our early stage app we had a huge mess with the Redis based queue system. It was one of the biggest sources of errors and a massive pain to troubleshoot or even to monitor what is going on. So we investigated in a bunch of different solutions including all the usual contenders, and we ended up with: Let's just ditch the messages / queues altogether for now and just do boring old cron like jobs invoking internal api endpoints in regular intervals. This made 9 out of 10 cases a lot easier to maintain, and for the remainder, while temporary more difficult, we introduced a queue again at a much later stage.

I'm not saying you shouldn't use and you can't benefit of RabbitMQ at small scales or any technology for that matter. But I think too often in tech decision making ones own or the companies perceived importance, what is cool or what would look good on a CV takes precedence over what really fits the problem in context.

Odd, I've been running it for years and it's been flawless. Integrating new clients is a breeze using their, or any amqp, library.

brew install rabbitmq?

I agree rabbitmq is often adopted in places it shouldn't, or ill-configured.. But to completely get rid of it because you didn't take the time to RTFM when setting it up seems a little extreme.

And redis/rabbitmq have completely different use-cases 80% of the time. Sounds like you were trying to get drunk on kombucha.

You are right in our case redis failed spectacularly with celery queue when tasks in queue exceeded the memory of the machine. It took weeks to diagnose as error message were inconsistent and setting up a Message Queue cluster with redis is a nightmare.

Went back again to tried and tested rabbitMQ and it works so well. Also adding new nodes and removing nodes so easy, just used ansible to setup Erlang cookie and connect the node (thanks to OTP and BEAM). The best part is for important task queue for which we cannot accept failure we built a mechanism for fault tolerance. When you work with high availability and fault tolerant queue rabbitMQ is so good, it can recover from hardware or even VM failures, can’t say the same for redis which was a nightmare even with redis cluster.

Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.

It must be emphasised that, despite the name, ZeroMQ is not a message queue. It is a networking library. The current blurb says:

> ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework.

And the old meme was "ZeroMQ is a replacement for Berkeley sockets".

It's a pretty cool networking library. But it makes no sense to think of it in the same slot as RabbitMQ, or even Redis.

That the GP mentioned it does make me wonder if they don't really understand what they're doing.

EDIT I love that the official guide actually has this diagram in! Pieter Hintjens's death was a sad loss. http://zguide.zeromq.org/page:all#How-It-Began

As an aside, the ZeroMQ Guide is the finest piece of technical writing I have ever seen - closely followed by Pieter’s (sadly unfinished) “Scalable C” book. His non-technical writing is also excellent.

I had a fun time implementing the Paranoid Pirate pattern with my coworker a few years back. He wrote the server part in Python, I wrote the client in PHP. We essentially built it as a wrapper to run some C code our boss wrote that we didn't want to write a PHP extension for - we used Python as a broker to allow for some concurrency. Worked super well.

a Redis list (what he's using) would handle this easily (I think Redis' PubSub also handles this)

Amusingly, RabbitMQ is the one message broker that HASN'T given me grief in my career.

With docker it takes 5 seconds to set a rabbitmq instance locally.

Your fault is using pika. I thought the same untill I tried some other python libraries . Try Oslo messaging . It's used in openstack (enough said).

We used both zeromq and its sort-of-descendant nanomsg but eventually moved to NATS because of it's approach to high availability (every node knows the topology of the whole cluster) and zero management overhead.

Some criticize NATS for the absence of message durability in the core technology, but we figured out we can drop this requirement in 95% of cases. Your microservices should be highly available anyway, so there's always a live consumer - and it's better to handle the message to it directly rather than introducing the overhead of storing a message somewhere and dealing with more-than-once delivery.

There's a bit more to that like you should let your microservices exit gracefully and finish processing consumed messages. And of course in 5% of cases, where you can't allow to lose a single message, you have to use NATS Streaming or the likes, but so far we've been greatly impressed by NATS for high load.

ZeroMQ compiles to well over 600 kilobytes of machine code (just on 32 bit x86).

  $ size /usr/lib/i386-linux-gnu/libzmq.so
     text    data     bss     dec     hex filename
   662134   12952      24  675110   a4d26 /usr/lib/i386-linux-gnu/libzmq.so

Yikes, is there an operating system kernel with virtual memory, USB support, a few filesystems and a TCP/IP stack hiding in there?

It's like 35% of glibc:

  $ size /lib/i386-linux-gnu/libc.so.6 
     text    data     bss     dec     hex filename
  1917549   11624   11112 1940285  1d9b3d /lib/i386-linux-gnu/libc.so.6

Which is itself heavily bloated, trying to provide as much POSIX as it can.

I'm equally surprised by your anecdote.

I found it quite simple to install RabbitMQ server and its admin panel in my WSL local dev environment.

And the cloud/prod instance took a few clicks (just spun up a DO Marketplace server image) followed by < five minutes of RabbitMQ user and firewall configuration.

It was also dead simple to start using RabbitMQ within my application. I found a well maintained package, installed it, edited a couple lines of my application's config, and everything just worked.

I specifically avoided Redis based on my understanding that it can't guarantee message persistence, so if it crashes, your unprocessed messages are lost.

I'd be interested to know what the 3 lines of python were and also more details about the redis server you deployed to replace RabbitMQ during the outage.

I'm using a bit of a hyperbole but using redis list as a task queue is a simple as:

    while True:
        msg = r.lpop(key)
        try:
            result = do_something(msg)
        except Exception as e:
            log.error(f'failure for msg "{msg}" got {e} back to queue {key}')
            r.rpush(key, msg)
            continue

Pop a member from a list, if failed plop it back to the end. It's simple, explicit and just works™

It just works, until PUSHing the member back fails. At least use RPOPLPUSH and a separate worker queue to make sure you don't accidentally drop packets.

Don't get me wrong, I'm using Redis queues myself and trying to get rid of the last remnants of RabbitMQ in our code, but there are use cases where RabbitMQ means less thinking about stuff, because it just works(tm)...

The other classic variant here is when msg is poisonous. Then pushing it back onto the queue means it'll go again at some point, poisoning your system again.

In practice, you need some dead-lettering with a message timeout. Something RabbitMQ does provide out of the box, for all its complexity.

couple of problems with this: (1) RPUSH can fail (2) LPOP's result might never reach your client after redis has executed it.

rabbitmq used to be one of the hardest things to self-host (even at relatively small scale). But it has been pretty stable since moving to Cloudamqp as a managed solution.

I had the same experience. I had particular trouble configuring encryption for connections. It didn't help that they'd switched configuration file-formats, but the documentation and tutorials still used the old format.

Never had any stability issues though. Once it was up and running, it was solid.