> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer.
RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol".
If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop.
If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck).
> If a certain message processing takes very long, all other messages in that partition will have to wait.
You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.