Hacker News Reader

Comment by tcoppi | Hacker News Reader

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware.

On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing discrete events in a reliable manner.

View on HN

I'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it:

https://engineering.linkedin.com/distributed-systems/log-wha...

An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common english usage.

IIRC, we started working on Distributed Log at least a year before that post.

The Twitter blog post in other comments says that "At design time we had concerns about Kafka’s I/O model...", which means that at the least Kafka was extensively studied during the design phase of DistributedLog, which may have started earlier than Kafka's release of course.

If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.

I agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.

Jay Kreps, architect of Kafka, calls it a log.

https://engineering.linkedin.com/distributed-systems/log-wha...

What does "boring" have to do with any use-case?

aka atomic broadcast

I don't think "distributed event processing framework" describes this system at all. It's exactly what it says on the tin: a replicated log (i.e. event log) storage system that one can build other systems on top of. It's not processing events as far as I can see?

You put events in one end, and it distributes them, reliably and consistently, to distributed end-points that can consume them. That is far more powerful than simply being used to log events for later retrieval, though it can be used for that. I have seen systems using Kafka that process tens of thousands of time-sensitive events a second and distribute them all over the world for independent processing as part of a larger system.

"real-time event journaling and dissemination system"?

Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.

Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.

When I think journal, I think a permanent record of rdbms transactions or perhaps even a blockchain. I agree with you though that the persistence of data needs to be defined when discussing a journal of any sorts.

You might prefer 'ledger' to emphasize the immutability of the journal.

"Log-structured file system" or "journaling file system" is often how such things are described. The notion being that the primary operation of interest is appending new data records, rather than modifying existing data.

https://en.wikipedia.org/wiki/Log-structured_file_system

The use of the term "log" in some contexts implies concepts like lossy, unimportant, non-durable, etc. However, in the context of log-structured file systems or journaling file systems, the sense that's intended is append-only as a primitive, atomic, consistent, durable (replicated) operation.

Kafka isn't that unique. There are several hosted options as well as commercial software that can do what Kafka does.

Kafka has the open-source/java community and ecosystem that fits in well with the rest of the current big data processing stuff though.

Reply on news.ycombinator.com