> when consumers are lagging behind, producer throughput falls off a cliff because lagging consumers introduce random reads

I am confused by this. The format of Kafka's log files is designed to allow reading and sending to clients directly using sendfile, in sequential reads of batches of messages. http://kafka.apache.org/documentation/#maximizingefficiency

> However, we can’t use Kafka as a queuing system because the maximum number of workers you can have is limited by the number of partitions, and because there is no way to acknowledge messages at the individual message level. You would need to manually commit offsets in Kafka by maintaining a record of individual message acknowledgments in your own data store, which adds a lot of extra overhead — too much overhead in my opinion.

I just want to clarify this - you're limited to N concurrent consumers for N partitions per consumer group.

If you have an O'Reilly subscription, you can also check out this detailed report comparing Pulsar and Kafka: https://learning.oreilly.com/library/view/apache-pulsar-vers...