> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y).

One solution would be:

Kafka allows you to easily create new streams from the "monolog" stream that normalise the data to a certain schema. Consumers can then just consume these new derived streams.

Another was mentioned in another reply (create a new stream that ultimately replaces the monolog).

> Document pipelines is a rare instance of a context where XML is the best choice.

XML does not really offer anything here that could not be achieved with tools that are nicer to work with? It's just a file format, basically. Why wouldn't Protobuf work? It also saves a huge amount of disk space vs. XML.

> Secondly, they should have a gateway coming out of the file store. For each downstream consumer, they should have a distinct API.

This is Kafka. They can have distinct streams for the consumers since you can always derive new kinds of streams.

> You shouldn't use it as a long-term data store.

Why not? Kafka has support for infinite retention and Kafka has very strong guarantees about always writing data to disk and not losing a single event.