We have implemented a similar pattern with https://github.com/wavii/darner and a daemon to take the messages and move them to AMQP out of band. It is a healthy pattern to send asynchronous messages to a local daemon before sending them out. Takes the external server out of the path of failure.
We do a similar thing using SysV Message Queues. Other than being operationally weird due to their age, SysVMQs are actually a really great fit for this problem.
I hadn't heard of Apache Kafka before. As branding goes, it's hard to do worse.
In my mind, Kafka is synonym with complexity, being obscure or byzantine, and as opposite to "useful" as I can imagine. See also: nightmare.
Maybe they were trying to channel the idea of the run on sentence which is somewhat more taboo in English than in German, and leads to the Kafka labored style, which he purposefully did. If I try to think of a positive angle, it is that the system can "keep going" perhaps? Anyway, all this stuff is great though.
Run-on sentences aren't taboo, they are just grammatically incorrect. Long sentences may be taboo, but that's a different thing.
Interesting. I really like the idea of using the UNIX sockets to be, essentially, a message queue before going to Kafka. This may be used on my next project.
There is a dedicated thread for monitoring the socket so clients don't block trying to write. In case of transient load spike or slowness in Kafka, the bulk of the queueing should occur inside the daemon, where you can see queue lengths via web interface. God luck with your project :-)