Hacker News Reader

Comment by wg0 | Hacker News Reader

I think SQS is cheap enough to build on as a messaging queue even if you're not hosting within AWS.

Out of the widely underrated AWS services include SNS and SES and they are not a bad choice even if you're not using AWS for compute and storage.

View on HN

The problem is rarely cost, it's operational overhead.

Using SQS for a queue rather than my already-existing Postgres means that I have to:

- Write a whole bunch of IaC, figuring out the correct access policies - Set up monitoring: figure out how to monitor, write some more IaC - Worry about access control: I just increased the attack surface of my application - Wire it up in my application so that I can connect to SQS - Understand how SQS works, how to use its API

It's often worth it, but adding an additional moving piece into your infra is always a lot of added cognitive load.

+. And then you have to figure everything one more time when you decide to move to (or to add support for) Azure/GCP.

You get exactly once when you consume with pgmq and run your queue operations inside transactions in your postgres database. I can't think of an easy way to get some equivalent on SQS without building something like an outbox.

SQS FIFO has exactly-once processing

That's not what the parent post was referring to. If SQS (or your ability to talk to it) is down and your database isn't, what do you do?

SQS is at least once PG can give you exactly once

Not sure why this is making people upset.

Because it's incorrect. If you have any non-postgres side-effect, you can't have exactly-once (unless you do 2PC or something like that). There isn't any technology that gives you "exactly once" in the general case.

That's not how exactly once is defined for queue. We are talking about semantics of what queue systems is providing.

Nobody will understand it like that.

Anyone who has ever selected queue service/product will understand it like that. Because thats one of the most prominent features that gets highlighted by those products:

SQS Standard queues support at-least-once message delivery.

NATS offers "at-most-once" delivery

etc.

SQS FIFO has exactly-once processing

well that's a stretch it has "5 minute window" You can hold a lock on a row in PG queue for as long as you need

pgmq (which is linked on this gist) provides an api to this functionality. It can be 0 seconds, or 10 years if you want. It's not a row lock in, which can be expensive. In pgmq, its build into the design of the visibility timeout. FOR UPDATE SKIP LOCKED is there to ensure that only a single consumer gets any message, and then the visibility timeout lets consumer determine how long it should continue to remain unavailable to other consumers.

Reply on news.ycombinator.com