First, we use them to allow app users to run (large) data processing tasks without keeping connections open for a long time. Basically letting our users start a data processing job on the client side and then let them continue doing something else, and not forced to stay until in the same page until the data processing job is completed. We check for status updates in the database; once our data processing job is complete, it updates the status of the job.
Second, in the backend, we have multiple tasks (compute resources) running that are ready to take on these data processing requests based on messages in the queue. To allow for bursts of user requests for data processing, we make available multiple tasks whose job it is to run the data processing jobs. We place data processing request messages in the queue, which are then picked up by any of the available (non busy) data processing tasks. With a local/internal queue you suggested, we’d have a harder time figuring out and ensuring the right tasks picks up the data processing job, and that only one tasks does so to avoid duplicate effort. Or we might have to have local queues on each task, and somehow coordinate them. With the centralized queue, we don’t have to worry about this - whichever tasks is available picks up the message and runs the data processing job. When the tasks takes the message it pops it from the message queue and no other task picks up the same message (unless we want it to for some other reasons).
With a queue you don’t need to know, and the recipients can subscribe to information on the queue themselves.
Especially useful in event-driven architectures.
Push vs pull.
There’s also a difference in making a command to another service vs event or state notification. In the case of a command you often want to wait and find out if it went ok. Events or states published onto a queue are more of a case of “here is a thing that happened, deal with it how you wish”
It's not a bad idea at all, you probably use message based systems every day without thinking about it, culturally however the idea is irresistible for enterprise developers who live for abstractions. Just something to keep in mind.
The webpage I wrote twenty years ago would accept your POST request, wrap up everything into the email, send it, return a redirect to a GET to the same page, and finally send you a page saying "Thanks! I've sent you an email with a link to click on to confirm it's really you." You would not get this web page because your browser would have timed out, or maybe you would but you'd think "Jeez this site is so slow" not realising that it's just waiting for you.
The webpage I wrote ten years ago would accept your POST request, wrap up everything into an email, send it to a message queue, and so on except maybe now I'd say "I am sending you" rather than "I have sent you", which you'd see immediately. Around the same time, a queue worker notices a new email in the queue, sends it, hangs around for a minute or so waiting for your server. You get the email, click the link, wow what a nice fast responsive site!
The webpage I wrote last year would do much the same thing, but the queue worker might flip a flag in your account to say the email has been sent but the link has not been clicked. Meanwhile a wee bit of javascript is polling an endpoint watching for the flag, and flips the text round to say "Check your email, it's been sent".
It's easier to handle certain situations, like if you have multiple recipients, you may want to notify all of them, at least one, at most one, etc.
It's also a way of handling consumers that may be unavailable for whatever reason and your source may not want to have to deal with that. For example, if the producer is a cron job, you may not want it to have to hang around for an arbitrary time if your consumers are already busy. Sure, this means that the queue is available, but in principle it's supposed to be more available than the consumers.
If your needs are fairly basic, that's probably the best approach. You also eliminate some overhead (piping messages through a middle man, operating said middle man, etc.).
But it's my understanding that if your needs are a bit more involved, the complexities of building a solid queue mean that it'll take time away from building your actual product. In that case, you may be better served by an off-the-shelf solution which already handles the corner cases and is good to go.
I think this is the usual decision between using an external product and rolling your own.
2) Reliability. Imagine, if a customer places an order in a store, but the server process crashes and the order is lost. The customer won't be happy about this.
A mesasge queue might help in both cases. Note that you might implement a queue in a SQL database as well, but this still will be a message queue.
I'd probably have started out with -just- the outbox though and bolted the HTTP request sending onto the database, then looked at switching to a dedicated queueing system later. But that's a "because I already understand the care and feeding of PostgreSQL" choice as much as anything else.
With the message bus the clients can (roughly) send requests as frequently as they want, while servers will handle them as fast as they can, without the danger of missing a request or having clients to wait. The queue is the "asynchronization" mechanism.
Secondly, it's decoupled from your application. Trying to do things directly works for a while, then you do an in memory array, then your application crashes or gets updated and lost those messages.
then you notice that you can do much more than that, but most people start here.