I may be describing the problem incorrectly, but I know vendors we talked to were aware of this issue and had workarounds; IIRC Aiven had tooling to easily spin up a temporary new "mirror" cluster for the new consumer to catch up.
I may be describing the problem incorrectly, but I know vendors we talked to were aware of this issue and had workarounds; IIRC Aiven had tooling to easily spin up a temporary new "mirror" cluster for the new consumer to catch up.
Partition count only limits concurrent consumption within a single consumer group. One consumer group won't impact another unless its consumers are doing sufficiently bad things to bottleneck the network or disk.
The default consumer read sizes are so small you will hit broker CPU and worker thread limits long before network throughput. (Both consumer batch sizes and broker threads can be increased trivially but there's not much documentation around when to do this.)
And fetching small amounts of data repeatedly doesn't impose much overhead unless you're deliberately disconnecting and reconnecting between polls.
And I assure you, you can definitely bottleneck network before CPU and/or network threads, Kafka was literally designed for very large numbers of consumers.
And as for tuning, Kafka The Definitive Guide is pretty much as the name suggests. I've been recommending it very strongly, especially the chapters on monitoring and cluster replication, for years.
You can download a free draft copy of the 2nd edition from Confluent, check it out :)
But max.partition.fetch.bytes is only 1MB.
1 MiB is allegedly the ideal batch size for throughput.
It sounds like you're asking if you can double the number of readers in a system with no performance impact. If you're at capacity, the answer is obviously no. Yes, every consumer takes some i/o and CPU on the brokers serving the data. I have never used Pulsar but I'm sure that's also the case there.
It definitely prioritises tail consumption over read from 0.