Amazon SQS is 15 Years Old
aws.amazon.com
aws.amazon.com
Like you can only send certain character sets. I had a very angry customer, and in the end it was some character (0x07?) that couldn't be sent via SQS.
I had to either be certain that contents would never contain an SQS-unallowed character, or base64-encode all my contents.
SNS and EventBridge is push and 1:many.
Kinesis and MSK is pull and 1:many.
Sure you can mix and match and get all confused. But I would stick to that.
[0]: https://medium.com/airbnb-engineering/dynein-building-a-dist...
I would never trust anyone who says this, regardless of who did what with it.
SQS Standard (which was the only for a long time) has historically been a nightmare for almost any use case including job queueing. I love creating a whole separate cache to ensure when SQS delivers message multiple times, that I don't act on them. That's just ONE issue that I don't need to have. When you get into FIFO to avoid that, now your costing is ridiculous at scale AND the (relatively small) number of issues with FIFO that are still outstanding .
eg https://tomgregory.com/3-surprising-facts-about-aws-sqs-fifo...
I second the parent post. I've billions of messages through SQS at a previous job and I can remember having issues with SQS availability exactly once (due to a systemic failure involving DyanmoDB where cascading failures took down almost everything (and all of our integrations) that would have hosed us completely if not for our durable-queue/s3-journal[0] at the edge). SQS is simple to use, does what it says, and has very good SDK support. A++++ Would build a business on top of again!
I've found its trivial to just have a unique ID in the message and just check if the ID already exists in a jobs table to avoid double processing.
I pretty much always have some metadata table needed for the job anyway, so its not like I'm building something extra.
I interpret arduinomancer's post above to mean that if the receiving service must action a single message once and only once (which is sometimes but not always necessary) then you need to give each SQS message a unique ID and that ID needs to be stored in as DB by the receiving service. I've used that pattern a few times and it works well (and doesn't result in unnecessary coupling between sender and receiver). In the past, I've used messaging systems that gave stronger transactional guarantees, but those systems were things like JMS or MQSeries and I don't really want to go back to those days.
I was referring to your (seemingly snarky, and I apologize if I mistook your intent) comment that you hoped I wasn't suggesting to a shared database between services.
I understand and agree that this is what OP was stating. What I was saying was that having the consumers and producers sharing a queue is not that much different from having them share a database, but a database gives you other guarantees as well. (I am a big fan of queues and have been using SQS since 2007.)
In other words, the database is still going to be your bottleneck/SPOF and a queue just introduces needless complexity; you might as well just write to the DB directly and have a table in the DB (or something like a Redis set) to track the jobs.
A queue with indeterminate ordering is better aimed at idempotent jobs than jobs that will eventually need to be serialized by necessity or design.
Apologies, I did write a snarky comment and I'm very sorry that I did that, rather than asking for clarification and/or explaining my position a bit more clearly.
I take the view that sharing an SQS queue (or a topic, or a stream...) between two services is OK. It creates a contract between those two services, of course, and part of that contract might be that there is a uniqueId field that the receiving service is going to have to persist in its own database in order to only action each request once, but it doesn't add an excessive amount of complexity or coupling between the services, not for me at least.
I'm still not OK with having two services share the same database, though, even if you're using it for a narrow purpose. For me, the danger is that you end up introducing more coupling than just agreeing on a message format and messaging semantics. I've been burned by that once or twice and I don't want to get burned by it again and I'm pretty happy that it's widely seen as an anti-pattern these days and personally I dogmatically avoid it. I do take your point that the database is going to end up being a SPOF/bottleneck, though, and I don't really have a great answer for that in the general case.
SQS (standard and FIFO) has another problem that precludes using it in one case I have; the 256KB message size limit. Amazon has a Java-only workaround (Amazon SQS Extended Client Library for Java) that will spool large messages to/from S3 -- so clearly my case isn't unreasonable -- but there is nothing for the general case.
For the most expensive tier: 40 cents per million API requests for plain SQS vs 50 cents per million API requests for FIFO SQS.
Like many AWS Services, it started off pretty basic or limited, but gradually and incrementally improved over time.
I started off pretty skeptical, but over time I've grown to be impressed with SQS and SQS FIFO.
It's just polite to write the meaning of acronyms.