Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds | Hacker News Reader

Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

github.com

58 pointswanshao23 comments

The Kafka API and S3 API have each become the de facto standard in the stream processing and object storage domains, respectively. Compared to other Kafka solutions, AutoMQ has embraced them more effectively. I believe AutoMQ has a long way to go.

It's a bit unfortunate IMO because Kafka while perfectly adequate for streaming often winds up with someone wanting to use it like a work queue which predictably ends horribly.

Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly.

There was Kafka-on-Pulsar until recently until the developers essentially made it non-OSS which is pretty unfortunate for Pulsar adoption which is already low.

Can you elaborate more on the part where using Kafka as a work queue ends horribly? Genuinely interested!

Essentially the problem is that Kafka doesn't have a way of managing acknowledgements on a per-message basis. This means that consumers from Kafka topics are assigned exclusive access on a per-partition basis and the consumer group manager only tracks acknowledged offset of each partition.

As such you end up with a few main problems. The first is head of line blocking, what this means is that if a consumer reads a message from a topic it's unable to process or will take an inordinate amount of time to process it can't move forward without potentially having to replay every message since the problematic message if it doesn't want to risk not replaying a message that wasn't processed correctly. Secondly it means that you can end up with hot partitions if the "cost" of messages isn't uniformly distributed across partitions because load isn't balanced across consumers, i.e there is no work stealing or other mechanism for other consumers to help out processing a hot partition.

Log systems with queue/subscription overlays like Pulsar and GCP Pub/Sub solve this by doing per-message acknowledgement (sometimes referred to as selective acknowledgement vs cumulative acknowledgement that Kafka does) usually by layering a persistent subscription abstraction over the top of the underlying log.

This is in contrast to pure queue systems like RabbitMQ, SQS etc that use a heap or mailbox approach where messages are simply emptied out as they are processed and don't share the log style struction of systems like Kafka.

So TLDR. If you use Kafka like a job queue you will end up in situations where queue processing gets stuck behind a single or patches of unprocessable messages.

The mitigations for it aren't pretty. They either involve building your own selective acknowledgement layer or a series of retry queues that messages are pushed onto using Kafka transactions with a final dead-letter queue at the end etc. Instead either wait for https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A... if you really want Kafka or use something that already does this, i.e Pulsar.

Agree that Kafka is best suited for stream scenarios. However, we also see Kafka being extensively used in online business scenarios.

Be interesting to see how well they do.

Most of the Kafka deployments I've seen in recent years have been using cloud versions e.g. AWS Managed Kafka.

So it's great that you save money on compute but it does mean you are now operating it yourself which then costs money.

Indeed, operating Kafka can be challenging and complex due to its nature as a stateful and distributed system.

However, AutoMQ has adopted a cloud-native architecture, offloading storage to EBS and S3(https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8...), eliminating the need for replication and rendering the Broker stateless, which simplifies operations significantly.

Seems nice, but are there any publicly available documentation to be able to evaluate its readiness and what languages are there libraries for? I may be blind but I cannot find any.

If you need to ask you aren't ready for something like this.

That said, it's basically Kafka with the topic-partition persistence swapped out. So the libraries etc are just the standard Kafka libraries because the frontend listener is unmodified.

Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream (https://github.com/AutoMQ/automq/tree/main/s3stream) library, thereby harnessing the benefits of cloud EBS and S3.

One thing that isn't made clear is when writes are acknowledged.

Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage?

If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated in a similar fashion to traditional Kafka storage?

Yes, acknowledgments for writes occur once the data is committed to the EBS WAL, with each write operation bypassing the cache via Direct IO. Data is then asynchronously uploaded to S3.

Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable:

- AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.

You might be interested in our strategies for managing different types of failures:

- In case of an EC2 instance failure, we take advantage of EBS's ability to be attached to multiple instances(https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes...). This allows us to quickly mount the EBS volume from the failed EC2 instance onto another Broker, facilitating a seamless failover process. - For failures that affect an entire availability zone, we utilize Regional EBS which is available in Azure and GCP: https://cloud.google.com/compute/docs/disks/regional-persist...

Ok yeah, multi-attach was the magic I was looking for to handle failure of instances.

Thanks!

jpgvm is right. AutoMQ didn't modify the computation layer of Kafka, but only revamped the storage layer. This means you can use AutoMQ just like Kafka. This is very user-friendly for those who are already using Kafka and applying tools within the Kafka ecosystem.

I bet that all message queues and log databases will support S3, as these types of data generally have a large volume and aren't as economically valuable (don't get me wrong, what I mean is that these databases won't be frequently read and processed).

Can't agree more! S3 will be the modern data storage primitive. Also, the move towards shared storage and separating compute from storage is a key trend in cloud-native architecture, enhancing scalability and cost-efficiency.

Is it truly possible for AutoMQ to achieve 'scale-in/out' within seconds?

This is interesting.

They're making a lot of claims and in the readme present zero measurements. The official Kafka distro is pretty bulletproof since 7+ years ago. It'd be great if they could include more information for the (apparently). initiated!

I am open to new things.. with at least a tad of supporting data attached :D Ymmv.

Hi, thank you for your interest in AutoMQ. If you have any questions regarding cost-related data, here is an analysis report for your reference: https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZ...

Thank you for your openness to new things. Your support is indeed very important to a new project. BTW, AutoMQ itself provides real comparison data tested on AWS, and the related code is open for users to reproduce these tests. For a cost comparison with Apache Kafka, you can refer to: https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aw.... If you care about performance data, you can refer to our performance white paper: https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkx....

Scaling Kafka has always been a headache for me, but seeing AutoMQ storing data entirely on S3, making Brokers stateless, no longer needing to migrate data for Kafka scaling. Awesome.

Reply on news.ycombinator.com