It takes effectively zero effort to maintain my redis instance in AWS, so I’m not sure you’ve got a real argument here…
Brief: yes, a specific datastore is usually better fit for the job, except that until your app requires more performance than Postgres can deliver if you already _have_ Postgres you might as well just stick to it.
Same for SQS - SQS is incredibly performant, but there is a whole lot of features it does not have which a PG-based queue system like good_job will give you out of the box. Just off the top of my head - priorities, separate queues, scheduling - and, do not forget, atomicity with your main transactional workload.
So while it is usually - when everything works great and the workload fits - not a big deal to run a specialized store, it can be more economical and simpler to just stuff everything into the DB until you outgrow that.
This is basically not true, is my point. There is no meaningful "problem" with throwing up a Redis instance in AWS, this just doesn't mesh with my experienced reality.
Failover when the node dies. Clustering for high availability?
Backups? For a cache, probably not, for a job queue broker, probably necessary.
Making sure your app deals with inserting into Redis on successful transactions and not when a transaction is rolled back.
Getting up and running can be fairly painless, staying running on all edge cases and handling partial failures is what gets you.
It’s just a nonsensical and unfair comparison. You can run a single Redis instance with normal rdb disk syncs and don’t ever update it for years on end without issue. Is that guaranteed resilient? Absolutely not, but that’s not the scenario in discussion. We’re talking about the context of a bootstrap/MVP scenario, not an enterprise setup.
I’d take a single-node redis job queue everytime over a HA citus/postgres cluster improperly acting as a queue.
Needing Transactional semantics for jobs alongside an application operation makes a lot of simpler queue/tool choices difficult.
For Redis though, the overhead is far more trivial than something like Kubernetes or Kafka, or even Elasticsearch or MongoDB
You don't need to learn your tools for doing simple stuff under normal circumstances. You need to learn them to do bespoke surgery on live data while everything is on fire and the customers are threatening to fire your company. Or better yet, so that you can avoid doing that altogether by anticipating limitations during design.
That being said, doing everything in Postgres is also going to bite you if you have moderate scale. This is really the same mistake again. Postgres looks like a big truck you can just load up and load up, until you wake up one day and there are cascading failures across all services that touch that database, because you wrote a dumb query that took a lock for 5 entire minutes while it did an HTTP request. It's robustness will lull you into thinking something is working well when it's actually barely working.
(Before you object, yes, it is a better idea not to have multiple services talk to the same database, I hear you. And no, you shouldn't ever hold a database lock while doing an HTTP request, believe me I know. These things can happen.)
I’ve been using Redis for nearly 10 years and it’s been a seamless and pleasant experience. Honestly it sounds like you’re taking your specific experiences and overgeneralizing.
I don't know the nature of the applications you've been working on those last 10 years, but it was more or less the main database for a high bandwidth, low latency service I was working on, also using Elasticache.
Problem spaces vary. If you're using it as a cache with modest load and consistency requirements, maybe you never need to understand it. But those sorts of requirements often creep & change out from under you.
So if you're saying, Elasticache did a good job of abstracting Redis, sure, I agree. If you're saying, there is no additional cognitive load to adopting a new service in your data path, because you don't even need to understand it - that puts a shiver up my spine, and makes me hear Pagerduty alerts in my head.
I don't feel designing for a guaranteed high-availability application was part of the discussion at all.
https://redis.io/docs/data-types/streams/
You can make good job queues out of this, combined with sharding or consistent hashing, for low(ish) latency applications. Each shard has a stream, they operate on data stored in Redis, and you pass them the key to this data over their stream.
But SQS is great, and a great rebuttal to the article. Totally easier to prototype a job queue that way than with pg, and you probably won't need to move off of it.
ETA: seeing your list of missing features now, that all makes a lot of sense. In my mind the biggest advantage of SQS is that it glues together all the other AWS offerings, so you can go AWS -> Lambda for an instant job queue (with concurrency limits, etc. so you don't blow your hand off - perhaps undermining the simplicity argument). But everything you're saying makes sense if your job queue needs any degree of sophistication.
- No multiple queues - No priorities - Practically no scheduling (the delay is very limited) - Creating and tearing down a queue takes a lot of time and the number of queues is subject to AWS account limits - The FIFO/LIFO semantics (remember about no priorities?) will bite you when you least expect
It does have great durability unlike Redis though and will scale to much, much larger queues in an easier way.
It's all normal stuff, by a long shot not the end of the world, but it is stuff that you need to do, and it is more stuff, and it can bite you if you come unprepared and "just clicked a few instances into existence last year".
And then there is monitoring, dealing with security issues, and upgrading.
It all multiplies.
Elsewhere people are debating whether Redis adds that much more overhead. I think they are missing the wider point. It’s about how many pieces you will have. 1 is simpler than 2. 2 is simpler than 3. 3 is simpler than 12.
Nowadays there are so many great specialized tools that do certain things really well. And for most use cases, you don’t need them. And when a time comes that you do start needed them, you add them then. And people will grumble about the idiot who implemented full text search inside of Postgres, while being completely blind about the 7 years of saved time NOT managing elasticsearch across 38 environments and quarterly upgrades.
You'll also need to code for two integrations (orm + whatever you're using redis with), which may be a solved problem or not, depending of your stack. And even then, still more complex than just postgres, and more error prone considering you'll either have to ignore enqueue reliability, or find a complex way around it.
But yeah I think this is more applied to the folks who decide to spin up a large service when they just need a messaging layer that their DB can solve for them. It has non insignificant cost.
Redis is pretty low maintenance tho in many configs (k8s, on a vm, or a service like on aws have all basically been zero maintenance for me).