Hacker News Reader

Comment by bananas | Hacker News Reader

I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems.

Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.

View on HN

Do you find that the message broker makes that implementation easier/better than just using a database table to store the incoming messages until they can be processed?

I've worked on enterprise products, off-the-shelf and in-house, which use message brokers, and every time I could not fathom how the extra complexity was justified. If the messages were just queued in a simple database table instead, the implementations would have been much easier to work with: easier setup and admin (it's just part of the existing database), easier to check and monitor (just select counts from the table), easier to inspect the pending messages and fix problematic messages. In all cases I get the feeling the architects were just looking for an excuse to use a new technology.

I can only see high-load scenarios as a possible justification, where the volume of incoming message data is physically too much for the RDMS.

Many message brokers do use or support using a database table to store the messages.

What they tend to add is ease of scalability and failover, and higher level functionality such as some of what the author of the article complained about, like automatically marking the message as available again if the worker that was processed it did not give any liveness sign within X seconds to deal with hung workers etc.

A database table is just a storage medium. The programming model is what is important. Some tasks don't fit the imperative model very well, some tasks have transactions open for days, some tasks need correlation across hundreds of different integrations, some tasks need to happen in guaranteed windows after initiated, some tasks need to be parallelised across lots of systems. These are a few cases I can think of.

From a current position perspective, we use MQs for quoting. Someone raises a quote, this is parallelised across 20 providers all with different integration methods, transformed back to a common object model, inserted as a single transaction into a database table, handed over to another system as an integration task which calculates risk, posted back from that to our system, a risk model is applied, the data is updated, the user is notified. This is all plugged into a 2 million line java and c# jumble that evolved over 15 years from a C++/COM nightmare.

We rewrote it with full test coverage in a month. We can add a new provider in an hour.

Imperative/table integration. No thanks!

People who distrust this methodology need to grab a copy of Enterprise Integration Patterns. There's a huge amount of wisdom in that book that puts Joe blogg's average dynamic language task queue to shame.

Without wishing to use the phrase, but enterprise software is about a hell of a lot of stuff people don't understand. This is probably 1% of it.

Reply on news.ycombinator.com