Happy to finally see LogDevice open. We have been working on this for years now.
I imagine you looked at other solutions before starting this. A distributed log is a fairly simple idea to understand (hard to implement) but what pain point is being solved?
Seeing that it is written in C/C++ - would it be that logdevice is optimised purely for speed and responsiveness?
It seems very similar.
In terms of function. LogDevice is similar to the core of Apache Kafka.
- There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes.
- Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distributed among broker nodes which helps a lot when there are many readers.
All of this is done while preserving the total ordering guarantee thanks the separation of sequencing and storage.
The operator could for example set a bigger node set size for logs that are known to have multiple consumers and require more IO capacity.
At facebook, we have use cases where a single consumer will need to replay a backlog of records in a log, sometimes hours or days worth of data to rebuild its state. We call this a backfill. Node sets allow the IO to be spread across multiple disks which improves backfill speed and helps reduce hotspots.
-- Adrien from the LogDevice team.
Curious where logdevice would be a bad decision..
In Kafka bulk reading is very cheap, the broker basically just calls sendfile() to send a file segment with compressed message chunks. On the other hand only the leader of a partition can serve requests, so you are often limited by bandwidth. It looks like LogDevice has to do a bit more work server side, but may be able to read from all servers with a replica.
Kafka stores more metadata in the record wrapper, like client and server timestamps and partition key.
There are client libraries for C++ and Python.
Operationally they look similar - both require a Zookeeper cluster, and both require assigning permanent ids to nodes.
It would be interesting to see some benchmarks comparing LogDevice with Kafka and Pulsar. That said, I suspect from the lack of buzz around Pulsar that Kafka isn't a performance bottleneck for most people using it.
Unfortunately it's hard to change something that already works. Most users don't hit the performance limits of their tools so they'll just continue Kafka if it's already running.