romange

It's exactly that. It gives us a little chance to fight against the Giants.

In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda Data and others. You are right that acronym is confusing - it's not Boost license, it's Business License. Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.

If I would choose another language it would be Rust. Why I did not choose Rust?

1. I speak fluently C++ and learning Rust would take me years. 2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto the standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve the mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible. (and yeah - there is zero chance I could write something like helio in Rust)...

Yes, VLL algorithm kinda solves the throughput problem. You will still have some issues with fault-taulerance - i.e. what you do if a machine does not answer you.

For intra-process framework it's not an issue (as long as we do not have deadlocks).

And no, it's not just because of io_uring it is faster. It's also because it's multi-threaded, has absolutely different hashtable design, uses a different memory allocator and many other reasons (i.e design decisions we took on the way).

we use io_uring for everything: network and disk. Each thread maintains its own polling loop that dispatches completions for I/O events, schedules fibers etc. Everything is done via io_uring API. All socket writes are done via ring buffer etc. If you run strace on DF you won't see almolst any system calls besides io_uring_enter

Yes, we use io_uring for networking and for disk. io_uring provides a unified interface on linux to poll for all I/O events. Re disk - we use it for storing snapshots. We will use it for writing WALs.

And we have more plans for using io_uring in DF in the future.

I do not know about aerospike, have never run their software but Scylla are the champions in what they are doing. Having said that, Scylla provide full durability guarantees, that means they need to touch a disk upon each write or update. Disk I/O is relativle expensive in the cloud and requires either large instances or special storage optimized family types like i3 etc.

Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.

I've been working on DF for the last 6 months and I implemented ~130 commands. It's still missing about 40% of the commands to reach parity with v5/v6.

I will continue working on DF. Primary/Secondary replication is my next milestone.

The fact that we based our core hashtable implementation on paper from 2020 does not justify it?

If I would choose another language it would be Rust. Why I did not choose Rust?

1. I speak fluently C++ and learning Rust would take me years. 2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve tha mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible.

I do not even know how to to that comparison. Redis and DF share the same protocol and the same API so it's each to compare with memtier_benchmark.

The reason for this is the networking limitations of each instance type. DF consistently reaches the networking limits for each instance type in the cloud.

On r6g it's 1.4M qps and then it's saturated on interrupts due to ping pong nature of the protocol. This is why pipelining mode can reach several times higher throughput - your messages are big. c6gn is network-enhanced instance with 32 network queues! it's the most capable instance in AWS network-wise. This is why DF can reach there > 3.8M qps.

You are correct - GET/SET do not require any locking as long as they do not contend and they do not in those benchmarks. You are right that for MSET/MGET you will see lower numbers. But still it will be much higher than with REDIS.

This is our initial release and we just did not have resources to showcase everything under different scenarios. Having said that, if you open an issue with a suggestion of a benchmark that you would like to see I will try to run soon...

Of course, I disabled. Why do you think it's suspicious? P.S. You can easily reproduce it - I wrote the command line and the instance type.

If you look below the throughput table you will see the 99th percentile latency of Dragonfly in the table.

Now, please take into account that DF maintains 99th percentile of 1ms at 3M! qps and not at 200K.

True. Keydb tackled the same problems as us. But we chose differrent paths. We decided to go for a complete redesign, feeling that there is a critical mass of innovation out there that can be applied for inmemory store. KeyDb wanted to stay close with the source and be able to integrate quickly with recent developments in Redis. Both paths have their own pros and cons.

memcached was born in 2003. Redis was born in 2008. Both have strength and weaknesses. DF tries to unite them into a single backend and keep the strengths of each one of them.

Thank you. And if you are curious to learn more - we would love to share! And we will.

Yes, helio is the library that allows you to build c++ backends easily similar to Seastar. Unlike Seastar that is designed as futures and continuations library, helio uses fibers which I think simpler to use and reason about. I've wrote a few blog posts a while ago about fibers and Seastar: https://www.romange.com/2018/07/12/seastar-asynchronous-c-fr... one of them. You will see there a typical Seastar flow with continuations. I just do not like this style and I think C++ is not a good fit for it. Having said that, I do think Seastar is 5-star framework and the team behind it are all superstars. I learned about shared-nothing architecture from Seastar.

Re helio: You will find examples folder inside the projects with sample backends: echo_server and pingpong_server. Both are similar but the latter speaks RESP. I also implemented a toy midi-redis project https://github.com/romange/midi-redis which is also based on helio.

In fact dragonfly evolved from it. Another interesting moment about Seastarr - I decided to adopt io_uring as my only polling API and Seastar did not use io_uring at that time.

Unfortunately unlikely. Shared nothing architecture works with messaging. So threads send each other messages like "give me the contents of that key" or "update this key". Operations like SET/GET will require a single message hop. Operations like RENAME require 2 hops. Transactions and Lua scripts will require number of hops as number of operations in the transactions. When it's in the same process - the latency penalty in negligible. But between different machines - you will feel it. But who knows, AWS already have very cool UDP-like protocol with very low latency... if this will become the standard inside cloud environments maybe we can create a truly distributed memory store that spans multiple machines transparently.

Dude, open an issue :) I was waiting for someone to ask for that because I do not know anyone who uses it :)

Yes, I can confirm that Memcached can reach similar performance as DF. However, one of the goals of DF was to combine the performance of Memcached with versatility of Redis. I implemented an engine that provides atomicity guarantees for all its operations plus transparent snapshotting under write-heavy traffic and all this without reducing the performance compared to memcached.

Having said that, DF also has a novel caching algorithm that should provide better hit rate with less memory consumption.

I honestly think it's impossible to reuse Redis OSS to make it multi-threaded besides what KeyDB did. Btw, I did reuse some of the code - but it's around single value data-structures like zset, hset etc.

With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++. My choice was using fibers - which provide more natural way of writing code.

I have not designed the distributed long thingy. Will do it in the next 2 months.

It's really good questions. I invite you to try it yourself using memtier_benchmark :) if you pass `--key-maximum=0` you will get a single key contention when doing the loadtest. Spoiler alert - it's still much faster than Redis.

It's not a simple sharing. It's actually share-nothing architecture and doing multi-key operations atomically is pretty complicated thing. I used paper from 2014 to solve this problem

ScyllaDB and Seastar was my insipration when I thought about the architecture for Dragonfly

Guys, I am the author of the project. Would love to answer any questions you have. Meanwhile will try to do it by replying comments below.

Includes a highly performant RPC engine. Implements a MapReduce framework for single-node, multicore applications. The MR framework works natively with Google Cloud. It's a first open sourced framework that I know of that combines Boost.Fibers and Boost.ASIO in a robust manner.