Comment by zmmmmm | Hacker News Reader

Comment by zmmmmm | Hacker News Reader

This reminds me more of Apache Camel[0] than other things it's being compared to.

> The process initiator puts a message on a queue, and another processor picks that up (probably on a different service, on a different host, and in different code base) - does some processing, and puts its (intermediate) result on another queue

This is almost exactly the definition of message routing (ie: Camel).

I'm a bit doubtful about the pitch because the solution is presented as enabling you to maintain synchronous style programming while achieving benefits of async processing. This just isn't true, these are fundamental tradeoffs. If you need a synchronous answer back then no amount of queuing, routing, prioritisation, etc etc will save you when the fundamental resource providing that is unavailable, and the ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. If you go into this ad hoc, and build in a leaky abstraction that asynchronous things are are actually synchronous and vice versa, before you know it you are going to have unstable behaviour or even worse, deadlocks all over your system and the worst part - the true state of the system is now hidden in which messages are pending in transient message queues everywhere.

What really matters here is to fundamentally design things from the start with patterns that allow you to be very explicit about what needs to be synchronous vs async (building on principles of idempotency, immutability, coherence, to maximise the cases where async is the answer).

The notion of Apache Camel is to make all these decisions a first class elements of your framework and then to extract out the routing layer as a dedicated construct. The fact it generalises beyond message queues (treating literally anything that can provide a piece of data as a message provider) is a bonus.

[0] https://camel.apache.org/

> The ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all.

Async frameworks don't eliminate the possibility of long running processes, that continue to process long after responding a request - this is still possible with specific libraries/frameworks, they'll only take away the synchronous interface and provide as asychonous one instead.

It is also important to note that error handling will be different between these 2 paradigms and it's important, whatever the most suitable one is, to acknowledge this since it forces us (developers) to handle the potential errors differently depending on the approach we choose.

I think you're stating exactly my point?

The pitch of MATS is that it let's:

> developers code message-based endpoints that themselves may “invoke” other such endpoints, in a manner that closely resembles the familiar synchronous “straight down” linear code style

In other words, they want to encourage you to feel like you are coding a synchronous workflow while actually coding an asynchronous one. You are pointing out that error handling needs to be different b/w these paradigms and you are correct, but that is only the start of it. A framework that papers over the differences is at very high risk of just creating a massive number of leaky abstractions that don't show up in the happy scenario but come back and bite you heavily when things go wrong.

(I'm saying this as a long time user of Camel which models this exact concept heavily and also experiences many of these issues)

Hmm. I want to distance this library pretty far from Camel!

Wrt. "papering over": Not really. I make it "feel like" you're coding straight down, sequential, linear, as if you're coding synchronously.

But if you look at the examples, e.g. at the very start of the Walkthrough: https://mats3.io/docs/message-oriented-rpc/, you'll understand that you are actually coding completely message-driven: Each stage is a completely separate little "server", picking up messages from one queue, and most often putting a new message onto another queue.

It is true that the error handling is very different. Don't code errors! You cannot throw an exception back to the caller. You can however make for "error return" style DTOs, but otherwise, if you have an actual error, it'll "pop out" of the Mats Fabric and end up on a DLQ. This is nice! It is not just a WARN or ERROR log-line in some log that no-one will see until way later, if ever: It immediately demands your attention.

I wrote quite a long answer to something similar on a Reddit-thread a month ago: https://www.reddit.com/r/programming/comments/1059jpv/messag...

> Hmm. I want to distance this library pretty far from Camel!

I'm curious what makes you distinguish it heavily from Camel? From everything you say it sounds to me like you are building Camel - or at least the routing part of it :-)

While these ideas was brewing, I ventured down into many libraries, Camel was one of them. It did not solve any of my problems.

Mats is messaging with a call stack. One messaging endpoint can invoke another messaging endpoint, and get the reply supplied to its next stage, with the state from the previous stage magically present: https://mats3.io/docs/message-oriented-rpc/#isc-using-mats3 and other pages.

Camel has, AFAIK, nothing of the kind.

It's interesting how we can have such different takes on it. Still everything you say maps naturally onto Camel for me.

Your example using Camel would look something like:

    from('rest:/api/myprocess') // using REST API endpoint as an example starting point
      .inOut()
      .to('activemq:main')
      .to('activemq:mid')
      .to('activemq:leaf')
    .end()

Then you would define the handlers:

   from('activemq:main')
      .transform { e ->
            return new State(...)
      }
      
   from('activemq:mid')
      .transform { e ->
            e.body.number1 = 
      }

All the processing stages are happening asynchronously and results are passed back and forth by Camel. If things error out half way through you get a printout from Camel of the whole workflow including each step, where exactly it failed and what the content of the state was at that point (and it'll throw the lost message onto DLQ if you want to re-do the process).

Mats does not explicitly define a set "route" through a bunch of stages. I view Camel more like a Workflow system, which is pointed out elsewhere in the discussion threads here - the workflows are defined external to the actual processing steps.

With Mats, you define Endpoints, which can be generic like a REST endpoint can be generic: "AccountService.getAccountListForCustomer" - which can be used (by invocation) by sevaral other endpoints.

Also, IIRC, the process you show with Camel is now effectively thread bound - or at least JVM bound. If the node running that process goes down, it takes all unfinished, mid-flow, processes with it. The steps are not transactional.

With Mats, the "life" of the process lives in the message. You can literally bring down your entire system from GCP, and rebuild it on Azure, and then when you start it up again, the mid-flow processes will just continue and finish as if nothing happened - as long as you brought along the state from the message broker (in addition to your databases). Viewed "from within the process", the only thing visible is that there was a bit more latency between one step and the next than usual. AFAIR, you cannot get anything like this with Camel. The idea is of course not the ability to move clouds, but that the result is exceptionally robust and stable.

> It immediately demands your attention.

it immediately demands the attention of the ops team that you've thrown all problems at while hoping that the magic queue never stops being magic.

while the client is sitting there going "WTF, where's my shit?"

Ops team?! We, the developers, are monitoring the DLQs: It is our mistakes, and we must fix them. Operations' only role is to keep the VMs the ActiveMQ is running on active, and the database it uses responsive.

If you used it in a synchronous fashion for a end-user, you are correct: He'll get a timeout - but the message will blink red on the DLQ, giving the devs exact info as to what has happened. As opposed to a 500 or something similarly bad, which probably in the best case was logged somewhere, and hopefully someone is tallying up the error codes every few weeks..

I fail to see how messaging is worse in this case.

I fail to see how it's better.

Sorry about that. If you have any questions that could shine light on things, I am happy to answer.

Reply on news.ycombinator.com