Hacker News Reader

Comment by raju | Hacker News Reader

Previous discussion on HN with 108 comments—https://news.ycombinator.com/item?id=25591492

Whoa, the top comment is really +5 insightful.

Yeah, it's a great comment. Since then things like Temporal have sprung up to try and address this state machine complexity of "accidental decoupling"

> Accidental decoupling is where you have a complex state machine encapsulating a business procedure with multiple steps, and it's coordinated as messages between and actions in multiple services.

That might need emphasis on "in multiple services."

Within the same service, a granular set of messages (events) can still be useful for auditing or creating good read-model "projections" of what happened.

I thought event sourcing as a data storage strategy and queues as an execution model were orthogonal concepts?

It's true that the messages (state machine transitions) don't have to be a durable source of truth, but there are similar arguments to be made for granularity.

I could add 2c. If you ever need to store some meta alongside the message in a DB. For example status, some execution history, etc. Then it's better to avoid MQ at all. Of course if you can scale DB access from workers and you can couple producers/consumers via the same DB. But it's the case for almost all applications TBH.

I mostly agree, but from a devil's advocate position, the downside is the likelihood that you end up reimplementing queue basics like retries, delay/scheduling, and of course, the essential transactional state flips without locking or perf issues.

In my experience, the downside to the queue is losing all the historical statistics/state that you get for free with a database table. You have to instrument all that stuff manually, since most simple queues are designed to be transient once messages are confirmed.

I usually end up with a hybrid: store a copy of the state in the DB (along with all the job data), and essentially use the queue to hand off an ID or something pointing to the DB. You can then run queries against the best-effort state recorded in the DB, but the queue handles all the at-most and schedule/retry logic I don't want to handcraft.

Fwiw, HTTP is a stateless synchronous at-least-once many-to-one request-reply message queue with metadata.

HTTP is a transport. It doesn't have any other properties besides you could make a request and hopefully get a response. Semantic is defined by actual client/server implementations with corresponding backend.

MQTT is a transport too, but its design facilitates a brokered pub-sub message queue. The transport implementation is effectively the queue, as far as the applications are concerned.

MQTT as a transport is a part of elaborate spec describing client/server state machine. There is nothing similar for HTTP.

The application doesn't care if it's "just a transport" or "part of an elaborate spec". Functionally there is no difference between an HTTP transaction and a message queue transaction, for a specific kind of message queue. You implement them literally the same way in an application; open socket, connect, send, receive, optional loop, close.

The point is that a message queue is just a concept. It can exist in many forms for many different use cases. You evaluate the type of queue and implement something that provides for its requirements. HTTP does that inherently for certain message queue types, and for others it requires more code.

Another example is the perennial meme of "PostgreSQL queue". A database isn't a message queue, it's a database! Yet people throw some crap into the DB and call it a queue. Because the application doesn't care how you implement the logical concept. HTTP, RDBMS, MQTT broker, whatever.

thanks for noting, 2020 should be in the title, I was about to comment that event bridge (which linked article notes is new) is a rebrand on cloud watch events which have been around since January 2016.

Reply on news.ycombinator.com