I need a dedicated message queue... Or do I? | Hacker News Reader

I need a dedicated message queue... Or do I?

yaketyhack.blogspot.com

3 pointskbenson6 comments

Have I read it correctly that you are writing a system to distribute jobs among real people? When a person has a queue for assigned jobs? Then this is totally not suitable for a message queue system (AMQP/JMS). MQ system is for distributing messages among software components at runtime. Specifically it is not designed for querying messages or doing other stuff typical for RDBMSes.

It starts as distributing jobs among real people, but eventually a good portion will be shifted to being automated. Additionally, new job types will be added that are automated from the beginning, and the quantity of those will be much higher.

That more expanded use case is what's making this even worth talking about, IMHO. If it was purely jubs for real people, I would definitely just use a RDBMS.

I should have made that a bit clearer.

So probably you have two separate subsystems of "jobs" distribution. Assigning to a person, querying and updating job statuses from some UI is very different from automatic computation of a result by a software component.

Is it? I'm trying to replace manual tasks, and I would like to do it in steps. First by implementing a joq queue that can be used in house by employees to drive what work needs to be done more effectively, then by trying to automate some of those tasks, and providing what the automation would have done as a guide for the employee, and finally when there's enough confidence that the automated routine is sufficient to the task, letting it take over without human intervention.

Assuming that the order in which tasks should be done doesn't change after they enter the system, there is no reason you can't use a message queueing system here. Obviously, you need software components that act as proxies for each user, but you need that in any worflow system.

Probably the best thing to do is to see if you can draw up a flow diagram that shows where messages need to flow to accomplish what you want. But it sounds like using a message queueing system as the main internal mechanism for communication.

I've decided to implement in an RDBMS, partly because this will allow me to become familiar with the concepts of message queues at a much lower level. I'm going to try to provide a semi-generic solution so the result is usable for more than my initial implementation, but I think this will provide the most benefit to both my employer and myself. I'll be intimately familiar with the core concepts, and will then know how to parse the feature/capability lists of the various implementation available, and will then be able to make an informed decision if I need to replace the system later.

I like how writing about it forces me to crytalize my thoughts, so I'm going to write it up[1].

[1]: http://yaketyhack.blogspot.com/2013/04/rolling-your-own-mess...

P.S. I've used a similar technique before in the absence of a reliable recommendation on a technology, due to unknown constraints or otherwise. The result of that was "Don't write your own ORM if you can avoid it (unless it's just to learn)". We'll see if the same result applies here...

Reply on news.ycombinator.com