you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself
You're suggesting we engineer something on S3 and aws-cli, while complaining about engineering something ourselves when AWS offers a perfectly good queue service that requires no engineering?
Uff. I'm going to buy a hut in the woods and live in it.
perfectly fine with using SQS, just it will have worse availability guarantees than S3 - people should understand tradeoffs
It is really hard to beat the cost/benefit ratio of S3.
a lot of mediocre engineers cannot swallow a pill that all their expensive work with hundreds of hours of overengineering could be replaced by a couple of AWS managed serverless services stitched together with a few mouse clicks or a single .yml file
sometimes it really does make sense just to pay someone else to solve the problem. not always, but not never.
all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!).
Try to beat that reliability guarantees with whatever you handrolled, and I bet you will never be able to beat the cost of S3, even match the durability, reliability, availability guarantees at any reasonable cost at all
from https://aws.amazon.com/s3/storage-classes/:
Key Features:
Low latency and high throughput performance
Designed for durability of 99.999999999% of objects across multiple Availability Zones
S3 API support sounds great until your costumer builds a system with an "S3 compatible object storage" product. Soon you discover that many "S3 compatible" solutions aren't actually that compatible when pushed.
Then it costs $92 / TB to get it out again.
Also S3 has durability guarantees but it's very difficult to do a durable transactional write to S3. Try it a few million times and see. The API is a defacto shitty standard.
These two facts are rather interesting when it comes to doing a restore from your supposed backup or wonder why consistency guarantees between external metadata services (DB) and what is in S3 don't always line up.
if it is for migration: it is one time cost that anyone can swallow easily if they decided to leave AWS for something else.
If your data is worth < $94/tb - it is really not worth pulling it out of AWS. Just let it sit there.
or just use cloudfront to download your data ($8.5/Tb)?
this checks out.
for latency sensitive you will probably need redis pub/sub or something in-memory
Kafka isn't the right choice for most things.
SQS, MQTT, NATS, rabbit if you're wanting a lot of admin are all better (plus the crap that azure and google make)
If you need fast response Kafka is a bad choice.
If you are okay going to multiple digits of milliseconds then there are simpler solutions.
The only reason to use Kafka is the ability to guarantee order. For everything else it's second place at best.
Kafka has been the slowest out of them that I've used and definitely more complex to use.
It’s also proprietary.
It's still harder to get wrong than Kafka.