Virtual Log-Structured Storage for High-Performance Streaming [pdf]

hal.inria.fr

48 points·teleforce··7 comments

While basically almost no real world application needs more performance than what kafka delivers, i love to see research on this topic.

hopefully at some point a more easy to use, non java, "cloud-native" distributed log with decent client apis shows up. (if anyone wants to help building it, please ping me. i am 10 years into building startups and companies with architectures ontop of kafka and pulsar)

Why non-java? What do you think is missing that you'd like to build?

Easier to use? Kafka is rather trivial to setup and scale to any size, even more so as they move away from zookeeper

Can't think of a more well behaved java app. Heap usage generally low, never seen GC issues and fast as needed

Your dream has arrived! Redpanda [1] is a non-Java, ZooKeeper-free streaming platform. It's Kafka compatible, so you can use any Kafka client, of which there are many.

Finally, and perhaps most importantly, it is substantially faster than Kafka.

Disclosure: I work there.

---

[1] https://github.com/vectorizedio/redpanda

The problem with Kafka is not Java nor ZooKeeper - it is that non-native (non-Java) clients are buggy as hell (last time I touched them), because everybody have to re-implement all the complex logic for each language (often multiple times). Do you have any plans for Redpanda-native protocol/client library?

Is this for write heavy workloads instead of ready heavy workloads? Some of the ideas seem similar to Virtual Consenus in Delos [1].

[1] https://blog.acolyer.org/2020/11/09/delos/

Source code here: https://gitlab.inria.fr/KerA/kera

Reply on news.ycombinator.com