Show HN: KafkaLite – A simple message broker inspired by Kafka

It seems inspired by the Apache Kafka project in that it organizes logs into named topics, but it is not networked, does not split topics into partitions and segments, and does not do housekeeping like deletion of old logs.

It looks pretty new and immature, for instance https://github.com/panyam/KafkaLite/blob/master/src/kltopics... reads from a potentially uninitialized field and could randomly fail (but will not propagate the error!). Have the tests been run with Valgrind?

Not bad, but it could definitely do with an introduction in the Readme describing its purpose.

In short the purpose of this library was to allow any application to embed an event broker with persistence so that it could consume events at its own pace. The purpose of this library is NOT to replace or compete with GCD or java.util.conc (which provide excellent primitives to do in memory queuing and processing of events).

The main use case that we wanted this for was message synchronisation. Consider a mobile messaging application that allows users (on mobile clients) to send and receive messages as well as do other CRUD ops on messages . To synchronise these events across all clients (say users in a channel) we would have to keep a connection (eg websockets) open to the server that would receive publish events to the client or poll for change events periodically by the client (of course the server would have to maintain its own event log - but a Kafka event sprayer that publishes to websockets is easily done).

In either case, when these events are received by the client they need to be processed as soon as they are received or the client would have to refetch them (harder to do on a non-polling situation as the state of what events have been consumed is expensive to record and maintain). Problem with having to process these events as soon as they are received is that there is no way to prioritise the events and there is no way to defer processing to a time convenient to the client. So with this library we have an append only log of events onto which we persist the events as we receive them (and because we can have multiple topics for each KafkaLite - KL - context) it allows us to impose prioritisation on which events are to be processed when.

I have not built in replication replication as it can be built on top as and when required. Also sharing of messages on a particular topic across processes while doable was not an immediate priority (would love to hear other use cases for this) so was eschewed. One other thing I would definitely like to do is provide a few sample producers that are platform specific (eg websockets based publishers, polling based publishers) but these seemed not generic enough (at least as of now) so I have not had a chance to investigate these further.

I would also like to apologise for any confusion with the name (especially to the gods of Apache Kafka!). As I had mentioned, this was only inspired by Kafka's original design (ie a stateless, fast and append only log). This library is written in C so as to be portable across any platform and not having to incur the cost bundling and invoking a JVM (clearly there is a lot to be done for this to be useable on the server side) - Hence the "Lite".

Again I really appreciate the feedback. Even though you see it when you read others content, with your own content you start with the bias of clarity as I did. Even more so if any of you would like help in using it in your own applications I would love to have a chat as to what features we could add to improve the robustness and usefulness.