AhmedSoliman

It’s a mixed bag of design ideas. There is definitely inspiration from LogDevice (disclaimer, I am one LogDevice designers) and Delos for (Bifrost, our distributed log design). You can read about Delos in https://www.usenix.org/system/files/osdi20-balakrishnan.pdf

Agreed.

Nothing prevents you from using your own data layer, but part of the power of Restate is the tight control over the short-term state and the durable execution flow. This means that you don't need to think a lot about concurrency control, dirty reads, etc.

"Virtual Objects" is a cool concept, the name might not reflect the power it brings though. Luckily, the documentation seems to explain it well.

I will let the generation Y kids know your opinion ;)

Maybe sending a PR would help?

We use Zookeeper primarily for the EpochStore. This is the abstraction that you can you use if you want to replace Zookeeper. It shouldn't be that hard as long as Consul offers the same guarantees as zookeeper.

It's a very different architecture and design. You can head to https://logdevice.io/docs/Concepts.html to learn more about how LogDevice works.

In terms of function. LogDevice is similar to the core of Apache Kafka.

Correct. LDShell in logdevice was the starting point of python-nubia.

I cannot give you exact numbers, but here are some information that might be useful: - LogDevice ingests over 1TB/s of uncompressed data at Facebook. This already has been highlighted in last year's talk in @Scale conference. - The maximum limit as defined by default in the code for the number of storage nodes in a cluster is 512. However, you can use --max-nodes to change that. There is no theoretical limit there. Each LogDevice storage daemon can handle multiple physical disks (we call them shards). So, If you have 15 disks per box, 512 servers. That's 7680 total disks in a single cluster. - The maximum record size is 32MB. However, in practice, payloads are usually much smaller. - Zookeeper is not (currently) a scaling limitation as we don't connect to zookeeper from Clients (as long as you are sourcing the config file from filesystem and not using zookeeper for that as well).

Hope that helps.

Happy to finally see LogDevice open. We have been working on this for years now.

We are looking for Scala contributors, if you are interested, please sign up and we will contact you for code access.

I +1 that, the tooling of Scala is comparable to what is available to Python and Ruby developers, given that building a proper IDE for such language is not a simple task, but right now Scala IDE (eclipse-based) and Intellij IDEA's support is more than satisfying to me.