I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.
That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper.
I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots).
I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-)
So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah.
BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
> The thing I don't like is that we also need BookKeeper, but yeah.
Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.]
> sirixdb
Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working.
Wish you success in your effort!
https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto...
https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"?
From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that.
Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.
I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning.
I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable.
I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant
Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.
One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties.
With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.