ActiveMQ: Not ready for prime time
goodstuff.im
goodstuff.im
We ended up writing a new queuing system from scratch in a month, and in its first week it was already more stable, performant, and bug-free than ActiveMQ.
This is why you should NEVER let an architect choose your software - ALWAYS the sysadmin who will actually be on-call to support it should have the final say.
Ah, if only that happened in the real world...
I just found a thread I started about ActiveMQ and scaling:
http://activemq.2283324.n4.nabble.com/6000-ActiveMQ-clients-...
Note the last message there. 36,000 threads when using NIO! And these answers were from the people commercially supporting ActiveMQ (FuseSource). I never did find out (on that thread or elsewhere) whether ActiveMQ has ever been used in a situation that large. I suspect it hasn't.
Ah, ok... gotcha. Didn't realize the time-frame.
I don't remember HornetQ.
It's fairly new... basically the successor to JBossMQ. It is reputed to be blazing fast though. I've been experimenting with it, but haven't used it in anger.
We investigated RabbitMQ, which seemed good but still a bit young to bet our company on.
Fair enough.
1. Do you need transactionality? What if your consumer experiences an error after it takes a message off the queue but before it finishes up processing it? Is it ok for that message to disappear forever? The common use case to consider is a consumer which takes a message from a queue, does some processing, and updates a database. To avoid distributed transactions, people often write these consumers so that they check first to see if a message has already been processed. This way, one simply ensures that his DB transaction commits before committing to the consumption of the message.
2. What level of message durability is required? Is it ok for all enqueued messages to disappear when a power cord is inadvertently pulled? I used ActiveMQ to populate a user behavior data warehouse. As users did stuff on a website, the application servers would enqueue observations (went to the product detail page, etc.) and these observations would eventually wind-up in a big Oracle DB. A delay of a day or so didn't matter, but we didn't want to lose more than a fraction of a percent of our observations. So, the queue was in effect a temporary system of record, and we had to allow for reboots, power outages, etc.
3. High Availability. Many modern queueing implementations can be deployed in redundant, scalable "meshes", ActiveMQ included. I haven't kept up on the feature sets of RabbitMQ, JBoss's offering, etc. or I would comment more here.
There are obviously a bunch of other considerations, but these are often the ones which are ignored when comparing ZeroMQ to ActiveMQ, etc. There's no right or wrong answer. "Lighter" implementations are often appropriate for messages where, if the shit hits the fan with your environment, the messages are useless anyway. IIRC, eBay builds pages by firing off a bunch of async requests for page parts, waits a maximum of N milliseconds, and then renders the page based on which pieces made it back on time. There's no value in persistence or transactionality, but the mesh sure better scale up.
No matter what your queue provider, you're most likely just using TCP sockets anyway, so don't think you need to be using Erlang just to take advantage of RabbitMQ.
Yes, this means the MQ can basically be thought of as a black box. But that's also kind of the point.
I'm sure my information is old, perhaps these things are improved in newer versions. Still, the behavior of the Erlang VM itself was relevant.
As a C++ programmer, I wouldn't feel comfortable abstracting away memory management for this type of server which is both so memory intensive and requires high reliability.
But of course many people swear by highly reliable JVM based stuff, so perhaps I'm wrong about it. :-)
It was true that previously RabbitMQ could run out of memory and be unable to take more messages, but that changed in 2.0 and it will now swap messages to disk. Of course, then you pay the consequences of slow disks and can always run out of disk space, but nothing is invincible.
Prior to 2.0, the contents of Rabbit's queues were always in RAM. Even durable queues with persistent messages always had their contents mirrored in RAM. As of 2.0, Rabbit can store queue contents on disk or in RAM, so as it faces memory pressure, it will send data to disk. If the disk is too slow for it to relieve its pressure, it will block connections that have sent data in the past until it can free up enough memory to continue.
Last I checked, Rabbit does by default keep the associative map between queues and message in RAM, so its memory usage isn't technically bounded (it's a around a dozen bytes per message, IIRC). If that's a problem, the toke plugin uses tokyo cabinet to store the associations, which I believe allows Rabbit to have entirely bounded memory use. I also haven't checked on that for a while, so it may be that toke is installed by default now.
Erlang was created by a telco in the 1990s and since then has been battered in production by serious users. It's real software. The version that Ericsson provide as open source is not crippleware as you seem to imply.
"RabbitMQ pays the performance cost of bouncing everything off the disk"
No it doesn't.
RabbitMQ only uses the disk if tell it to do so, eg if you require messages to be persisted when they cannot be delivered immediately.
"memory usage can grow and grow"
If you stuff data into a messaging server without draining it on the consumer side, then memory usage will grow. This will also happen if you write your messaging system in C++.
There are two solutions to this problem: - flow control, where you tell producers to back off - paging to disk, where you flush data from memory when it is on the disk
Neither of these is trivial to implement which is why there is a big gap between toy messaging systems and serious products.
As others point out on this page, RabbitMQ has support for both these features. In particular a lot of memory management capability has been added since 2.0.
I hope this helps.
In brilliant contrast, RabbitMQ is it-just-works software. It's a piece of infrastructure that works so well it practically just fades into the background. I couldn't imagine having to worry about a service in which the chief goal is to help alleviate reliability concerns.
Reddit are atill using RabbitMQ and as far as I can tell they are happy with it.
If anyone has any doubts or questions about this, please email us. info@rabbitmq.com
I can't believe the ActiveMQ/FuseSource guys weren't willing to work it out with David either. Perhaps there's a bit of open source/commercial software wrangling behind this story. Won't be the first/last time.
The BQL plugin, which is now sadly unsupported by the Rabbits, used to be a nice command line interface, it is a shame it is no longer supported.
Btw, I'm using RabbitMQ, and I love it. My needs do not include high load or high availability though so I can't speak for that. So what's nice about it? AMQP (you automatically get lots of tools, docs, "expectations", etc), very friendly and active mailing lists, small & clean code base, small footprint, simple, active (more features are always being added).
What I don't like about RabbitMQ? While it's FLOSS inside out, its development isn't exactly a "community" work. For example, I can't report an issue, attach a patch, and receive a reviewer/committer feedback about it and possibly get it in. In fact, I can't even report an issue into their issue tracking system -- I just have the mailing list. That said, I believe they said they are going to fix that part "soon".
But it's true that the community has been much more involved in clients like Pika, for example.
And yes we are planning to open up the bug tracker.
A piece of advice for anyone doing an open source project - start with an open tracker, because opening up a previously closed tracker is a royal pain in the butt.
(N.B. we need contributors to sign a contributor agreement.)
David (rabbiteer)
You mean there's a correct configuration? Also, Tomcat, Jetty, PostgreSQL, and Ngnix don't "install correctly" out of the box as described here. Tomcat clusters set themselves up? Postgres? Hell, I think default shared buffers allocation is 32 MB...
> Anything that streams bytes
I'm not sure what the author is trying to do. Doesn't sound like queuing to me...
The documentation is terrible, so I'll give that up. It is unfortunate that there is so much experimentation involved in setting it up. ActiveMQ is immensely versatile and configurable, and it's sort of necessarily complex to get it perfect for your task. It's not a turn-key software; it's a systems architecture component.
Perhaps the author is using a packaged version of Tomcat from his OS distributor.
That it's not production ready is pretty big claim. I don't want my clients reading this and getting all jittery on me.
I have to admit that I'm not a big user of ActiveMQ, but I've used a closed-source equivalent from a slightly bigger vendor (IBM) for many years. It also is not configured correctly out of the box for most enterprises. When new features or platforms come out they are often buggy. The documentation though is usually decent until you really want to dig into the internals and understand it.
Excuse my ranting -- I'm a middleware system administrator and WMQ is the necessary evil at my workplace.
We have had incidents where it's Master/Slave replication continues on failure when configured to shutdown the master and slave brokers. It doesn't achieve the consistency it lists as a feature, and provides zero visibility of progress. I feel many of the features the project lists are incomplete, and are listed as a grasp at straws to have the bigger list.
Due to the poor quality of the replication, we had to invest a tonne of time to implement replication at the block device level with DRBD+GFS2, complicating the system and (benchmarks pending) likely decreasing performance, all to achieve a 'feature' ActiveMQ boasts.
I would also challenge a project to have worse, more incomplete documentation.
At the same time, I thank the ActiveMQ team for their contributions and ask the community to not turn it's back on this potentially decent solution. The ideas are solid. It isn't production ready, I feel, but that just means it needs some love.
Cheers,
Tim Vaillancourt
ActiveMQ with active-backup setup over shared disk mount is the current choice, and the start-up is really slow if the queue has a lot of data.
RabbitMQ does not persist messages in HA fashion, so I've ignored it so far. Maybe HornetQ needs some attention.
I see a lot of flexibility and feature-richness in the queue landscape and it perplexes me that getting this simple combination of basics right is so difficult.
The docs for HA are here: http://www.rabbitmq.com/pacemaker.html
This also works with Veritas if you use that instead of DRDB.
Please note that we are currently QAing a new HA model which is active/active. Watch this space!
For example the last time I checked you can't block a produced in RabbitMQ based on the number of messages on the queue.
"one interesting concept about queues is contention and this is where RabbitMQ and others are behind"
Can you explain what this means? What kind of contention are you talking about. You say "RabbitMQ and others" - which others? Who implements this feature and what does it look like?
"you can't block a produced in RabbitMQ based on the number of messages on the queue"
Yes you can.
Well - it depends on your use case. RabbitMQ enables you to determine queue length, and supports flow control.
But, perhaps you had something else in mind?
- RabbitMQ has a memory based flow control based on the total memory you have in the machine. I know I can develop something over RabbitMQ to accomplish what I want (limits based on number of messages) but I prefer to have the support within RabbitMQ.
- WebLogic JMS can block producers based on number of messages as a flow control method.
In a later, simpler, project I had issues with ActiveMQ locking up after a certain number of messages, which was solved with an upgrade.
On a whole, I think the quality of ActiveMQ is lower than other open source projects with similar brand recognition, e.g. other popular Apache projects.
i think you could sum this up that he simply wasn't working in a situation that warranted the division-of-labor and configuration flexibility that the product offers.
another way of saying the same thing, which illustrates the cultural differences:
- activemq is intended to be used in "the enterprise". it tries to implements a logical ideal, which is a reliable infrastructure that services can use without being coupled to each other - without worrying about whether messages were received, or exactly who they go to. to reduce complexity it uses a central broker (so you send messages to a central "hub").
- zeromq is intended (imho) for programmers that want to wire things together. it's less concerned with abstractions and more with providing something simple clear, simple and flexible that can be understood and used well. to reduce latency it uses direct connections between peers.
from that viewpoint, you can see that the two are both orthogonal and yet similar... (disclaimer: i haven't used either, but i used work on an ESB so have a vague grasp of what's going on. please someone correct me if this is wrong - i might as well learn as i lose karma ;)
ps rabbitmq is somewhere in the middle and was (i think) originally more performance-motivated (i believe it's used in finance for example - when speed might be critical).
RabbitMQ's main motivation has been to make it easier to join systems together, scale your applications and manage complex environments. That is what messaging is for. Back in 2006, we felt there was a need for a good, stable and scalable open source licensed product that could compete with the incumbents.
Notice that I did not mention performance. RabbitMQ has good performance and it is used quite a lot in finance, but the majority of users are what you might categorise as "anyone using MySQL or Postgres".
Re activemq vs zeromq, I recommend reading "broker vs brokerless" on our blog.
Hope this helps.
They pass messages. They're fairly related.
It's also nice in applications like Mongrel2. Make an http request, in your browser, start up your web server, and the request completes! Great for development, where sometimes you switch to the browser faster than your app can restart after a change. With Mongrel2 (and thanks to 0MQ), you don't even notice the race condition.