I reckon your message broker might be a bad idea
programmingisterrible.com
programmingisterrible.com
Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself.
Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick.
Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem.
Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load.
You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work. It seems to be considerably more involved than doing it with HTTP. Basically, using a message queue pushes a concern which should be external to the worker into it, and that creates work that needs to be done.
I am coming to think that whilst it's okay for apps to send messages to a queue, they should never receive consume messages from one. Rather, i suspect there should be some small infrastructural component which pulls messages off a queue and sends them to workers with HTTP, via a load balancer. I think this component can follow some simple, generic rules, and leave the smarts in the load balancer as normal. I haven't fully fleshed this idea out yet, though.
How is load balancing anything but an implementation detail, since one can usually be written as a service utilizing the same protocol in question? That's how HTTP reverse proxies usually work (notwithstanding other methods of balancing load such as DNS).
Other monitoring aspects such as feedback are already built into well-designed MQs. If that feedback isn't being acted upon properly, then that's again a problem with the implementation -- the same could be said for a HTTP-based task queue that simply ignores errors. It's not "extra work"; it's just different.
depending quite what you mean here, ActiveMQ supports this.
"With an MQ, no component of the system has a global overview of load."
except the MQ surely? Again, ActiveMQ supports multiple consumers on one queue and handles balancing the messages to workers at the rate they require.
AWS SNS can deliver a notification to an HTTP/HTTPS end point. If your design permits the use of a hosted notification service then I highly recommend you explore SNS.
And yes, I do agree with all the concerns you have raised! In my company we use Kestrel with a couple of consumers. Load asymmetry and worker health check is a real pain for us.
But if you genuinely have more messages than the workers can process, pretending you don't is not solving a problem. With the broker you have the choice to decide what size of queue is acceptable without building your own queue logic into the client. The only case where HTTP is generally preferable there is the case where a queue of any size is always unacceptable: If you need the message processed right now, or not at all.
> Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself.
I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.
> Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick.
With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken - why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?
> Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem.
What is the difference between monitoring for 500 errors and monitoring for errors from an MQ? Something needs to generate those errors. Either you are monitoring the logs, or your app is sending responses directly. If you need/can use synchronous responses, either you are abusing a message broker for something it shouldn't be, or your app should be sending reply messages that the client needs to wait for (which may still be preferable if the processes are potentially very long running). There's no difference here.
> Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load.
With workers pulling work out of a message broker, they self-regulate: The ones on the highest loaded machines will pull messages slower. Load is usually the wrong thing to worry about from the app perspective - processing rate is much more relevant, and a metric that's trivial to make available from your app. Latency is easily handled by imposing strict limits and killing / pausing workers that miss them.
> You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work.
Frankly, in most of your scenarios I feel the HTTP alternative is what poses the extra work. You're imposing extra steps to manage adding workers to the load balancers; you're adding complicated load monitoring instead of letting the workers self regulate which ones have capacity to process the messages just by virtue of which ones pull the next message first.
Brokers are not always the right choice. They're generally the wrong choice if you want synchronous, low latency end to end processing. But for many systems, different parts of the system may process different types of jobs at different rates, and with different scale out requirements. In those cases, introducing brokers means each component only has to worry about their specific task, as the overall data flow management gets separated out in a separate component monitoring the brokers and taking appropriate action (such as, for example, spinning up more VMs to run brokers at some point in the pipeline).
I'm not sure what you mean. Are you talking about regulating messages going from clients to the broker? I was thinking about regulating messages from the broker to the workers. I haven't seen a broker which will let you regulate the flow on a per-consumer basis like that.
> I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.
It's one less hassle in the control plane, and one more in the worker. I would prefer to have this stuff in the control plane rather than the worker, since it's not something directly concerned with the work being done. Separation of concerns and all that.
> With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken
Yes. Exactly. I would like the system to do something sane in the face of broken workers.
> why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?
That's quite a bit more work than just taking it out of the pool. And killing the server isn't necessarily the right answer - if the instance is locking up because it's going a full garbage collection, for instance, the best thing is probably to let it finish and then put it back in the pool.
> What is the difference between monitoring for 500 errors and monitoring for errors from an MQ?
500 errors are thrown straight into the client's face. The client itself can log that something went wrong. The closest analogue with an MQ would be for the broker to log when a consumer rejects a message. Can you configure them to do that? I honestly don't know. You'll struggle to have the logs tell a story which makes as much sense, though, because the MQ doesn't have the context that exists in the sending app.
Don't get me wrong, i don't think message queues are worthless. There are problems for which they are indisputably the right solution. But i think people should be aware of the operational challenges in using them. I have seen a lot of naive enthusiasm for queues which doesn't seem to take this into account.
Industry knowledge of how to distribute loads has improved significantly in the past 10 years. I wish the author would touch on some other options like using a load balancer with consistent hashing[2].
[1] http://martinfowler.com/bliki/CatastrophicFailover.html [2] https://en.wikipedia.org/wiki/Consistent_hashing
Furthermore, a good broker supports some kind of failover. We run a RabbitMQ cluster behind a load balancer, and it works great - once e.g. we had a hardware failure, and the applications did not notice anything from it. Of course we also have to monitor the queues on the broker (one of the few ways to bring down Rabbit is to fill up its disk). But having a central broker for connecting apps also means you have a central point you can keep an eye on to check where your bottlenecks are (which queues build up).
Loosely coupled applications are also much more unix-like in the sense that you can make them do one thing instead of baking in your custom restart logic (over a network? please don't).
It's really quite trivial to ensure your load balancer is not SPOF if you're ok with clients having to reconnect on failure, and only slightly more hassle if you want to allow connections to survive one of your load balancer boxes failing.
I don't disagree with the OP that this is a common mistake, but the solution offered is more of a "we also made this mistake, so clearly it is a problem with MQs and not us!" which isn't actually true.
Additionally, MQs are really (IMO) a better fit for systems that either have a measurable level of complexity on writes or are lightly written but heavily read. You see the same general pattern failure with people who insist that RDBMS is The Answer(tm) even though the data they write/read isn't relational and often read-heavy.
If the consumer simply checked if the retry count was above a threshold, and then redirected the message to a dead-letter-exchange, this would solve the "stuck message" problem.
As for tracking build up of messages, this is just a case of adding something like a nagios check on size of queue.
It seems like the solution outlined in the post is just implementing by hand the features of a decent message broker they chose not to use.
Sounds like they managed to get the job done regardless.
At least, that's the conclusion i came to when i looked at redelivery for an application i work on that uses RabbitMQ. Please do correct me if i'm wrong.
The spec has a SHOULD for aborting after a retry count, but it doesn't appear to be exposed to clients.
Whereas every message queue system I've ever seen has incredibly complex documentation, and you have to study it for quite a while to even figure out what the message broker's model of the problem domain actually is.
I don't know enough about message brokers to say for certain they wouldn't be right for certain really complicated commercial sites with extremely high load, where you can afford a full operations team with one or two people who specialize in understanding message broker technology, but for anything smaller than that, to me it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology.
Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.
Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.
I've worked on enterprise products, off-the-shelf and in-house, which use message brokers, and every time I could not fathom how the extra complexity was justified. If the messages were just queued in a simple database table instead, the implementations would have been much easier to work with: easier setup and admin (it's just part of the existing database), easier to check and monitor (just select counts from the table), easier to inspect the pending messages and fix problematic messages. In all cases I get the feeling the architects were just looking for an excuse to use a new technology.
I can only see high-load scenarios as a possible justification, where the volume of incoming message data is physically too much for the RDMS.
What they tend to add is ease of scalability and failover, and higher level functionality such as some of what the author of the article complained about, like automatically marking the message as available again if the worker that was processed it did not give any liveness sign within X seconds to deal with hung workers etc.
From a current position perspective, we use MQs for quoting. Someone raises a quote, this is parallelised across 20 providers all with different integration methods, transformed back to a common object model, inserted as a single transaction into a database table, handed over to another system as an integration task which calculates risk, posted back from that to our system, a risk model is applied, the data is updated, the user is notified. This is all plugged into a 2 million line java and c# jumble that evolved over 15 years from a C++/COM nightmare.
We rewrote it with full test coverage in a month. We can add a new provider in an hour.
Imperative/table integration. No thanks!
People who distrust this methodology need to grab a copy of Enterprise Integration Patterns. There's a huge amount of wisdom in that book that puts Joe blogg's average dynamic language task queue to shame.
Without wishing to use the phrase, but enterprise software is about a hell of a lot of stuff people don't understand. This is probably 1% of it.
This is not because you do not understand how message broker works that they are not simple. Message brokers implement "reliable, well-understood" protocols, and you do not need to have engineers who "specialize in understanding message broker technology". This is downright FUD. The little complexity of those protocols is here for a reason, and, seriously, we're not talking Enterprise Java complexity.
It's just like saying that you could use bash scripts to automate your infrastructure unless you have Google scale, because Puppet, Chef, or Ansible are too complex and over-engineered.
Writing a database-backed RESTful service for a message queue is not without reminding me of those who reimplement RDBMS over MongoDB because RDBMSes are "over engineered".
If the AMQP protocol is too complex for you, there are other, simple protocols that message brokers implement.
But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it.
Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)
Can't you have a message broker AND embrace those design principles... without pushing responsibility onto your crawler? What happened to the Single Responsibility Principle?
Note that I have nothing against inexperience - I used to be too. And noobs are malleable - in that sense they're easier to work with than experienced people.
- Use a broker that supports a timeout on the messages. Many do. E.g. beanstalkd lets you reserve a message for a specified time period, otherwise it goes back on the queue automatically. You can let the server knows if you need longer to finish. But a locked process will not hold on to a job.
- Use a timeout on the crawlers. A hard timeout like alarm() that causes the process to die if it fails. Run them under a process monitor that'll respawn them.
- Have the crawler or a separate system monitor queue depth, and slow down / alert/ throw errors if the queue goes above a certain depth.
All of this is trivial.
If introducing a broker takes the whole app with it if its fails, then you're doing it wrong: Unless you need in-order guaranteed delivery of messages, making a message queue resilient to failures is trivial: Just add more of them, and let the client talk to all of them. If you need in-order delivery, then a active-passive setup is easy enough too. Put the queue in a synchronously replicated database if you can't afford to lose any of them. So his argument about failures, is an argument against letting "inexperienced programmers" set up systems with single points of failure, not against message brokers.
The irony of this is that his solution is that it's great when the process generating the messages to queue can be run on a single, reliable system, by someone who will notice when that system fails and recover from it. But you've now stuffed your crawler full of state, and created a central point of failure. Just what he complained about.
The broker solution is great when you want scale - you can multiply the number of brokers. You can multiply the number of machines the workers run on. You can multiply the number of machines processing the crawled documents. It is also great when you want resilience for the same reason: By decoupling, it makes it easier.
If you don't need that scale, then go for the simpler solution. But if you do need that scale, then his solution isn't a solution at all.
If you introduce a broker, and your queues start growing out of control. Guess what? You have a scaling problem you would likely have without the broker too. Only in a decoupled system you have a much easier way out: Add more machines/VMs with workers; add more brokers.
Don't feel you have a way of controlling it? This is what your monitoring and orchestration systems should be doing. Queues over a certain threshold? Trigger an alert, and optionally spin up an exra VM.
No, describing it in English on HN is trivial. Getting all of this to work in the really real world is incredibly difficult and time-consuming. Components and infrastructure will inevitably find novel ways of failing that your three-bullet architecture didn't take into consideration.
That said, I agree that this is a problem that can be solved with a message broker. I just don't think it is easy. If you can make a system work with fewer moving parts, then you probably should.
"Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour."
I reckon the blog author should really reconsider the black on dark-grey color scheme of the blog; contrast is not your enemy when it comes to a simple, text-focused blog. This might look pretty decent on whatever system it is that he is writing it on, but on my PC monitor (which is quite a good one, carefully color calibrated and brightness reduced for editing photos), it is a complete disaster.
(What, your browser doesn't let you turn off stylesheets? :-)