How did you solve producer fencing mechanism which was built into Bookkeeper with Kafka (Assume I need to limit only one client to be able to write to a topic)?
How did you solve producer fencing mechanism which was built into Bookkeeper with Kafka (Assume I need to limit only one client to be able to write to a topic)?
In Bookkeeper, one producer can fence the current active producer from appending to log. If majority of Bookkeeper nodes respond with OK to a FENCE request, then the current producer can not append to the log. This helps implementing master-slave database replication quiet trivial. Master keeps appending to log, and slaves keep reading from log. In case master fails, one of the slaves sends a FENCE request to Bookkeeper nodes which reliably blocks current Mal-functioning master from appending to db log.
However in Kafka, this isn't a native operation. Any client can push to a topic. There is no easy way to safely do this.
I was looking forward to see how Twitter team actually handled this specific use case.