stolsvik

Mats does not explicitly define a set "route" through a bunch of stages. I view Camel more like a Workflow system, which is pointed out elsewhere in the discussion threads here - the workflows are defined external to the actual processing steps.

With Mats, you define Endpoints, which can be generic like a REST endpoint can be generic: "AccountService.getAccountListForCustomer" - which can be used (by invocation) by sevaral other endpoints.

Also, IIRC, the process you show with Camel is now effectively thread bound - or at least JVM bound. If the node running that process goes down, it takes all unfinished, mid-flow, processes with it. The steps are not transactional.

With Mats, the "life" of the process lives in the message. You can literally bring down your entire system from GCP, and rebuild it on Azure, and then when you start it up again, the mid-flow processes will just continue and finish as if nothing happened - as long as you brought along the state from the message broker (in addition to your databases). Viewed "from within the process", the only thing visible is that there was a bit more latency between one step and the next than usual. AFAIR, you cannot get anything like this with Camel. The idea is of course not the ability to move clouds, but that the result is exceptionally robust and stable.

:-)

As mentioned a few places, the actor model is actually part of the inspiration behind Mats: https://mats3.io/background/what-is-mats/#attempt-at-a-conde...

Yes, as far as I have come to understand, REST and similar protocols like gRPC are definitely the most common inter-service communications solutions.

Witness e.g. Netflix's stack with Hystrix, and the popular Kubernetes add-on Istio service mesh with its Envoy service proxy. And as the sibling comment points out, Kubernetes' own control layer is REST.

Messaging is an alternative, which I believe is underused due to its completely different mental model which is harder to reason about. Mats tries to make that simple - by "emulating" that you are working with something like REST, by introducing "Messaging with a call stack" and the state object.

More here: https://mats3.io/docs/message-oriented-rpc/, here: https://mats3.io/background/what-is-mats/ and much more here: https://mats3.io/using-mats/endpoints-and-initiations/

edit: Btw: "async/promised based" - this is not the point. It is true that you can code REST with "async/await" or Promise-semantics. However, you have still then bound the process to a piece of memory on a server. If the server goes down, all the processes it held, be it blocking in a thread, or in some Promise object, goes down with it. With Mats, the process "lives in the message". As long as you got it onto the message broker (i.e., the initiation went OK), it will never disappear: It will either go to completion, or it will pop out into a DLQ (Dead Letter Queue), where it can be inspected, you can jump to your logging system to find its processing, and you can - importantly - reissue the message and thus restart the flow from where it stopped, if the underlying problem is now cleared (e.g. a database that was down, or some data that (due to a bug) was in some erroneous state which now is fixed)

I write about that here: https://mats3.io/using-mats/endpoints-and-initiations/, search for "A comment about the meaning of asynchronous".

You can read about the MatsBrokerMonitor, a Mats-specific tool for inspecting the message broker: https://mats3.io/docs/matsbrokermonitor/. It gives more details about Mats flows than e.g. ActiveMQ's standard inspection GUI gives. I should however really work on that text, and throw in some images. You can clone and run the development-server if you want to see it in action: https://github.com/centiservice/matsbrokermonitor/blob/main/...

Mats has two levels of priority: "ordinary" and "interactive".

JavaDoc for interactive: <https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiat...>

Notice that this flag follows the entire Mats Flow, so that the initiated flow gets a "cut the line"-card for all queues it goes through.

> they are using an "async" system to simulate a thread-oriented system with blocking. You can do that, but why?

The "simulation" is purely visual, or rather "cognitive load reduction"-wise. It is explained multiple time throughout the pages om https://mats3.io, for example here: https://mats3.io/docs/message-oriented-rpc/, and here: https://mats3.io/using-mats/endpoints-and-initiations/ (with multiple examples, both pure Java and Spring-based definitions, also comparing to an ordinary REST controller).

The clue is to be able to use the simple mental model of sequential steps, with "RPC calls" intertwined. However, this is only on a superficial level, to make the developer's reasoning simple and familiar - what really happens is that your Mats Endpoint is really multiple stages, where each stage is an independent little message processor. To achieve this, Mats implements "Messaging with a Call Stack", where you get a state object which "magically" follows you through the stages, simulating the stack variables you'd have if it actually was a proper method.

It works surprisingly well.

> That's a synchronous send message and wait for reply system, like REST.

This you get if you employ the MatsFuturizer: https://mats3.io/docs/sync-async-bridge/ This is a "tack on"-tool to the otherwise fully async nature of Mats/messaging.

> For this to really work well, the message passing has to be integrated with the CPU dispatcher

It sounds like you are 100% set on speed. This is not really what Mats is after - it is meant as a inter-service communcation system, and IO will be your limiting factor at any rate. Mats sacrifices a bit of speed for developer ergonomics - the idea is that by easily enabling fully async development of ISC in a complex microservice system, you gain back that potential loss from a) actually being able to use fully async processing (!), and b) the inherent speed of messaging (it is at least as fast as HTTP, and you avoid the overhead of HTTP headers etc.

It is mentioned here, "What Mats is not": https://github.com/centiservice/mats3#what-mats-is-not

I believe this is detailed a few places on that website, e.g. here: https://mats3.io/docs/message-oriented-rpc/

“Messaging with a call stack”, “invokable, message-based async endpoints”

:-)

I actually mention the actor model as inspiration for Mats here: https://mats3.io/background/what-is-mats/#attempt-at-a-conde...

> If your queue is the bottle neck (yes it happens) how do you know there are more nodes getting added to it? How do you rebalance the topic over multiple new nodes in the queue?

In a message queuing system, there is no rebalancing - you just add nodes, which then also get messages, typically in a round-robin fashion. This is not Kafka.

But of course, you can max out your message broker. As you can with a network switch for that matter.

> oh look i got an error and retried a few seconds later

So now you need a retrying mechanism. Which is provided as a basic feature of a messaging system. And with the transactionality, you do not get the problem of wondering whether the first attempt went halfway through, or not.

Not sure when you saw the HN post, but the original title was "Why messaging is much better than REST for inter-microservice communications" - @dang / HN changed it.

The article specifically points out that this is for inter-service communication. However, you can also use the same system for synchronous communication out the your end user clients, then using the MatsFuturizer: https://mats3.io/docs/sync-async-bridge/

Thanks, I'll take that Erlang-comparison as praise!

(I actually mention the actor model as inspiration here: https://mats3.io/background/what-is-mats/#attempt-at-a-conde...)

> message queues are just shared DBs with some extra ordering

Completely agree. And I have stated that elsewhere in these threads. I mention it here: https://mats3.io/using-mats/matsfactory/#connecting-to-the-c...

And I have a feature-issue that explores implementing Mats using DBs: https://github.com/centiservice/mats3/issues/15

While these ideas was brewing, I ventured down into many libraries, Camel was one of them. It did not solve any of my problems.

Mats is messaging with a call stack. One messaging endpoint can invoke another messaging endpoint, and get the reply supplied to its next stage, with the state from the previous stage magically present: https://mats3.io/docs/message-oriented-rpc/#isc-using-mats3 and other pages.

Camel has, AFAIK, nothing of the kind.

Ops team?! We, the developers, are monitoring the DLQs: It is our mistakes, and we must fix them. Operations' only role is to keep the VMs the ActiveMQ is running on active, and the database it uses responsive.

If you used it in a synchronous fashion for a end-user, you are correct: He'll get a timeout - but the message will blink red on the DLQ, giving the devs exact info as to what has happened. As opposed to a 500 or something similarly bad, which probably in the best case was logged somewhere, and hopefully someone is tallying up the error codes every few weeks..

I fail to see how messaging is worse in this case.

The original title was "Why messaging is much better than REST for inter-microservice communications" - @dang / HN changed it.

I feel the change is wrong in at least two ways: The linked article literally argues that messaging is much better ("superior"), and it is specifically arguing for the use in inter-service communications: "The arguments here are for service-mesh internal IPC/RPC communications. For e.g. REST endpoints facing the user/client, there is a “synchronous-2-Mats-bridge” called the MatsFuturizer"

You can read about that bridge here: https://mats3.io/docs/sync-async-bridge/

As mentioned here: https://mats3.io/background/system-of-services/

.. Mats is meant to be an inter-service communication solution.

It is explicitly not meant to be your front-facing endpoints. If you are DoS'ed, it would be from your own services. Of course, that might still happen, but then things would not have been much better if you used sync comms.

It is true that you can bridge from sync to the async world of Mats using the MatsFuturizer (https://mats3.io/docs/sync-async-bridge/), but then you still have your e.g. Servlet Container as the front-facing entity.

(Also check out https://matssocket.io/, though)

Hmm. I want to distance this library pretty far from Camel!

Wrt. "papering over": Not really. I make it "feel like" you're coding straight down, sequential, linear, as if you're coding synchronously.

But if you look at the examples, e.g. at the very start of the Walkthrough: https://mats3.io/docs/message-oriented-rpc/, you'll understand that you are actually coding completely message-driven: Each stage is a completely separate little "server", picking up messages from one queue, and most often putting a new message onto another queue.

It is true that the error handling is very different. Don't code errors! You cannot throw an exception back to the caller. You can however make for "error return" style DTOs, but otherwise, if you have an actual error, it'll "pop out" of the Mats Fabric and end up on a DLQ. This is nice! It is not just a WARN or ERROR log-line in some log that no-one will see until way later, if ever: It immediately demands your attention.

I wrote quite a long answer to something similar on a Reddit-thread a month ago: https://www.reddit.com/r/programming/comments/1059jpv/messag...

Hmm. Maybe not. But they sure have much in common: You define a set of things that should be done, triggered by something - either a schedule, an event (oftentimes a repository event, but it doesn't have to), or from another Github action.

Thank you!

Well. My point is that messaging inherently have all these features, without needing any other tooling.

The combination of transactionality and retrying is hard to achieve with REST, don't you think? It is actually pretty mesmerizing how our system handles screwups like a database going down, or some nodes crashing, or pretty much any failure: The flows might stop up for a few moments, but once things are back in place, all the flows just complete as if nothing happened. I shudder when thinking of how we would have handled such failures if we used sync processing.

The one big deal is the concept of "state is on the wire": The process/flow "lives in the message" - not as a transient memory-bound concept on the stack of a thread.

Thanks a bunch! Seriously. And I agree that AGPL is pretty harsh - I have a feeling that this is typically used in a "try before you buy" situation, where there is a commercial license on the side.

To make a point out of it: This is not event based in the event sourcing way of thinking. It is using messages. You put a message on a queue, someone else picks it up. Mats implements a request/reply paradigm on top ("messaging with a call stack").

In the interactive, synchronous situation, you do not "wait for an event" per se. You wait for a specific reply. When using the MatsFuturizer (https://mats3.io/docs/sync-async-bridge/), it is extremely close to how you would have used a HttpClient or somesuch.

MatsSocket: The Dart/Flutter implementation is used in a production mobile app. For the Norwegian market only, though.

The JS implementation is used in an internal solution.

Would have been really nice with a bit more usage, yes. It is actually pretty nice, IMHO! ;-)

I was referring to the "not open source". I edited my comment to be more specific.

To your latter part: This is exactly the point: Using messages and queues makes the flows take whatever time it takes. Settling of the mentioned orders are not time critical - well, at least not in the way a request from a user sitting on his phone logging in to see his holdings is time critical. So therefore, if it takes 1 second, or 1 hour, doesn't matter all that much.

The big point is that none of the flows will fail. They will all pass through as fast as possible, literally, and will never experience any failure mode resulting from randomly exhausted resources. You do not need to make any precautions for this - backpressure, failure handling, retries - as it is inherent in how a messaging-based system works.

Also, if a user logs into the system, and one of the login-flows need a same service as the settling flows, then that flow will "cut the line" since they are marked "interactive".

Hahaha, that's rough! :-)

I'll tell you one thing: "What the hell does this actually do?!" is extremely spot on! I am close to amazed to how hard it is to explain this library. It really does provide value, but it is evidently exceptionally hard to explain.

I first and foremost believe that this is due to the massive prevalence of sync REST/RPC style coding, and that messaging is only pulled up as a solution when you get massive influxes of e.g. inbound reports - where you actually want the queue aspect of a message broker. Not the async-ness.

I've tried to lay this out multiple times, e.g. here: https://mats3.io/docs/message-oriented-rpc/, and in the link for this post itself.

You are correct, this is meant as an AWS/GCP/Azure preventor. ElasticSearch situation. That is, AFAIU, the intention of the license I adopted. The "examples" part spells it pretty directly out, as I also try to do here: https://centiservice.com/license/

You may definitely use it anywhere you like.

We clearly do not agree.

Wrt. sync processing when using Mats: https://mats3.io/docs/sync-async-bridge/

But my better solution is instead to pull the async-ness all the way out to the client: https://matssocket.io/

Also, I have another take on the SOA failure, mentioned here: https://mats3.io/about/

It was definitely not because of async, at least as I remember it.

Relying on the e.g. Servlet Container not being able to handle requests seems rather bad to me. That is a very rough error handling.

We seem to have come to the exact opposite conclusions wrt. this. Your explanations are entirely in line with mine, but I found this "messy" error handling to be exactly what I wanted to avoid.

There is one particular point where we might not be in line: I made Mats first and formost not for the synchronus situation, where there is a user waiting. This is the "bonus" part, where you can actually do that with the MatsFuturizer, or the MatsSocket.

I first and foremost made it for internal, batch-like processes like "we got a new price (NAV) for this fund, we now need to settle these 5000 waiting orders". In that case, the work is bounded, and an error situation with not-enough-threads would be extremely messy. Queues solves this 100%.

I've written some about my thinking on the About page: https://mats3.io/about/

I guess you'd say the same about cloud functions and lambdas, then? To which I agree.

Paying per message would require the message cost to be pretty small. Might want to evaluate setting up a broker yourself if the cost starts getting high.

Cadence is a workflow management system. As is Temporal, Apache Beam, Airbnb Airflow, Netflix Conductor, Spotify Luigi, and even things like Github Actions, Google Cloud Workflows, Azure Service Fabric, AWS SWF, Power Automate.

A primary difference is that those are external systems, where you define the flows inside that system - the system then "calling out" to get pieces of the flow done.

Mats is an "internal" system: You code your flows inside the service. It is meant to directly replace synchronously calling out to REST services, instead enabling async messaging but with the added bonus of being as simple as using REST services.

But yes, I see the point.

"Not everybody is Google"

These concepts has worked surprisingly well for us for nearly a decade. We're not Google-sized, but this architecture should work well for a few more orders of magnitude traffic.

Also, you can mix and match. If you have some parts of your system with absolutely massive traffic, then don't use this there, then.

Note that we very seldom use "fire and forget" (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging.