Comment by westurner | Hacker News Reader

Comment by westurner | Hacker News Reader

Parent Full thread

Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework.

The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-...

Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol.

RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia

SQLite is written in C and has lots of tests because aerospace IIUC.

There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http

celery/kombu > Transport brokers support / comparison table: https://github.com/celery/kombu?tab=readme-ov-file#transport...

Kombu has supported Apache Kafka since 2022, but celery doesn't yet support Kafka: https://github.com/celery/celery/issues/7674#issuecomment-12...

I don't get where you are pointing at.

RabbitMQ and other MOMs like Kafka are very versatile. What is the use case for not using SQS right now, but maybe later?

And if there is a use case (e.g. production-grade on-premise deployment), why not a client-side facade for a production-grade MOM (e.g. in Celery instead of sqs: amqp:)? Most MOMs should be more feature-rich than SQS. At-least-once delivery without pub/sub is usually the baseline and easy to configure.

I mean if this project reaches its goal to provide an SQS compatible replacement, that is nice, but I wonder if such a maturity comes with the complexity this project originally wants to avoid.

When you are developing an information system but don't want to pay for SQS in development or in production; for running tests with SQLite instead of MySQL/Postgres though.

SQS and heavier ESBs are overkill for some applications, and underkill for others where an HA configuration for the MQ / task queue is necessary.

Tests might be a valid use case, but doesn't seem to be the goal here and there are some solutions out there if you dont want to start a lightweight broker in a container.

Why would you want to use the SQS protocol in production without targeting the SQS "broker" as well? The timing and the AWS imposed quotas are bound to be different.

There are plenty brokers that fit different needs. I don't see the SQS protocol especially with security and so on in mind as a good fit in this case.

The switching cost from local almost-SQS to expensive HA SQS for scale and/or the client.

SQS is not a reliable exactly-once messaging protocol like AMQP, and it doesn't do task-level accounting or result storage (which SQLite also solves for).

Apache Kafka > See also: https://en.wikipedia.org/wiki/Apache_Kafka

SQS is not particularly expensive.

I don't know where you are getting at. I work on a project with SQS and I worked with RabbitMQ, ArtemisMQ and other MOM technologies as well. At least once delivery is something you can achieve easily. This would be the common ground that SQS can also provide. The same is true for Kafka.

Reply on news.ycombinator.com