Guys, I am the author of the project. Would love to answer any questions you have. Meanwhile will try to do it by replying comments below.
Also as a Redis replacement, it's not clear what durability is offered, and for most Redis use cases this is close to the first question
It does sound like extendible hashing might have downsides in some scenarios also.
I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.
Note: I never use kernel modules due to the issues rhel/debian and I have had with such things in the distant past.
I'm not sure that concern is justified. It seems io_uring was pushed as part of the 5.1 linux kernel release, and Ubuntu 20.04 LTS seems to have been shipped with 5.4.
https://packages.ubuntu.com/search?keywords=linux-image-gene...
Also, a quick Google search pointed to io_uring patches for Ubuntu 18.04 LTS.
Of course, it is also possible there are situations where it doesn't perform as well.
The tradeoff the way I see it - one needs to implement 200 Redis commands from scratch. Besides, I think DF has a marginally higher 50th percentile latency. Say, if Redis has 0.3ms for 50th percentile, DF can have 0.4ms because it uses message passing for inter-thread communication. 99th percentiles are better in DF for the same throughput because DF uses more cpu power which reduces variance under load.
Re-durability - what durability is offerred by Redis? AOF ? We will provide similar durability guarantees with better performance than AOF. We already provide snapshotting that can be 30-50 faster than of Redis.
If I recall ScyllaDB has some excellent examples of demonstrating this particular tradeoff visually. A simple option would be a scatter plot where X = latency, Y = load or similar, with points coloured according to the system under test. Probably there is a better option, but this would likely be enough to sell me at least
op r6g c6gn c7g
set 0.8ms 1ms 1ms
get 0.9ms 0.9ms 0.8ms
setex 0.9ms 1.1ms 1.3msCurious: Why BSL? Why not open core [0] (or xGPLv3) like what most other commercial OSS projects seem to be doing?
Accomplishes the goal of preventing a cloud provider from stealing customers, but also ensures customers don't get caught in an "always tomorrow" trap when the deadline comes and the company realizes it only hurts them to fully share it.
Seems to align all interests pretty nicely.
(I'm as big of an OSS supporter as anyone, but we can't pretend we still live in a time where Google / Amazon / modern-Microsoft don't exist)
F/OSS means a very specific thing. If one can't possibly build a rocketship business, that isn't F/OSS fault. Of course, you've got an alt-movement in response to OSI and FSF's rigid adherence to its principles, speared on by tech companies who (think they) got burnt by other tech companies.
There's a lot of room between {completely F/OSS, that a cloud provider can implement and bankrupt the company supporting project development} and {completely evil closed source company like Oracle}.
In the end, we all want good software, with the maximum amount of permissions and source, free. It's just a question of tweaking the support model to get there.
IMHO, the "Don't call yourself OSS if you're not 100% OSI OSS" is counter-productive. But I understand why they do it, and I understand the history and abuses that caused them to do it. I'd just say if we no-true-Scotsman our approach in a way that precludes profitable, sustainable OSS companies... we're going to have less quality OSS. :(
Still, I really appreciate that you didn't choose a copy-left license.
On the license front, what is the "change license" clause listed? It says something about changing in 5 years. Does this mean it will become Apache licensed in 2027? Why would you put that in there?
In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda Data and others. You are right that acronym is confusing - it's not Boost license, it's Business License. Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.
However, I'm sad that instead of going with an Open Source license that protects against that, you're using a proprietary license. That alone is a nonstarter for many users, not because they want to compete with you but because they want to protect themselves and make sure they have a firm foundation to build on.
Much of the software you're citing as examples moved from Open Source to proprietary, harming their users in the process, and causing many users to seek alternatives.
So, if this were AGPL and I have an closed webservice that uses this, it is in the clear.
The performance improvements are not worth the legal/compliance overhead of adding non-FOSS to my stack. Much less so if the performance improvements are due just to some optimization choice in the underlying system. In the next 5 years, it will be easier to have Redis adding the io_ring optimizations than for this new project to become uniquely better.
I have a SaaS [0], my own (AGPL, by the way) open source project [1] and also work have the occasional contract job. It's much easier to say "just use Redis because it is FOSS and it gets the job done" across the board then to try to special-case the tools based on licensing requirements.
[1]: https://hub20.io
Absolutely not. First of all, you can always use any software under AGPL as released without any limitation.
Secondly, if you were to make changes and release them you can still run it any way you want.
Thirdly, if you are making changes for internal use, and for some reasons your really want to keep them secret, you can STILL run the database to your heart's contents as long as you don't provide it as a service to the public.
AGPL is much more permissive than people think. It's just providing a degree of protection to developers and and users from patent trolls and other uncooperative entities.
There really aren't that many popular AGPL server software to start with though, so it's an unfairly biased question.
From their FAQ:
> The market is quickly moving to consume most software as a service. This is a time of incredible opportunity for open source projects, with the potential to foster a new wave of great open source server side software. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture all the value but contribute nothing back to the community.
HN discussion from 2018: https://news.ycombinator.com/item?id=18229452
FAQ: https://www.mongodb.com/licensing/server-side-public-license...
Today, MongoDB is an AWS partner: https://aws.amazon.com/quickstart/architecture/mongodb/
The most recent notable example of open-source software going closed because of Amazon specifically is Elasticsearch, which switched from Apache to SSPL. The announcement blog post was titled "Amazon: NOT OK - why we had to change Elastic licensing".
HN discussion from 2021: https://news.ycombinator.com/item?id=25833781
Here [A]GPL is Schrödinger's license. At the same time people complain that it's too restrictive and not restrictive enough.
If you want to completely prohibit any SaaS usage closed source is the only option, like BSL. And I'll stay away from your software.
Also, smaller companies like to play safe. If FAANG with their million lawyers aren't touching it, why should I take the risk.
This is a first. If anything, it seems clear that a FLOSS license such as AGPL achieves exactly the opposite: is highly protective of the users' best interests, both in the short term and long term.
Could you elaborate on why do you think that users are harmed by standard, run-of-the-mill FLOSS licenses?
However, if you are a 'user' in the sense of a company wanting to make software to sell as a service, it doesn't protect you very much, and can make you vulnerable.
*Assuming it's not a memory-store as a service, obviously.
However this is rather uniquely due to Google's monorepo infrastructure. Google quite literally has all its products in the same source code repository, being built and statically linked together to create a single system binary (or so I am told--I've never worked there). In this case AGPL would virally infect the rest of your code and you could find yourself in trouble. Even if they don't, they might get hit with a discovery request for the source code of some project to prove compliance, and in the process have to provide their entire monorepo and all its business secrets to the court. Etc. Etc.
But these are concerns which stem directly from Google's monorepo architecture. It's not generally true of the whole industry. And yet Google's fear of copyleft--rational though it may be in their particular instance--has been copied throughout the entire industry. 99% of the companies out there have nothing to fear from the AGPL. But hey, if you're some random in-house lawyer at company XYZ, who are you to question Google's legal precedent?
That does not meet the definition of "user", does it?
In addition, if that was the motivation behind that absurd claim, why not be honest and just say "we don't release our software under a FLOSS license because what we actually want is to keep it proprietary"?
I could convince a legal team to allow an AGPL service, depending on how we were using it. It would be difficult to get them to allow us to use this license, and I definitely couldn't convince them to allow us to contribute to software using this license.
Effectively, you've ensured you won't have a community, because in reality the project is closed source, and dead if your company dies, which means it's not an option for me to consider.
I hope people will understand the reason behind bsl. I hope people will see apache there and will help us to grow. Based on HN comments some already do.
I understand the emotions people have regarding licenses that do allow them to work with software as they please - but I also think it’s useless to argue around licensing of a software instead of focusing on the good it can do. Something being proprietary has NO impact on your choice to buy it when outside of the software world. Why does it here?
I do have practical first hand experience with both. I can get a legal team to approve the use of AGPL, but I cannot get them to approve BSL, unless we've signed a contract with the company.
Having an understanding of the license, and some first hand experience seems like it would be a pretty important thing, when choosing it, especially for a business.
BSL is at least making an attempt to address the legitimate concerns of businesses -- both sides -- in a balanced and pragmatic way. I have no dog in this fight, being neither a BSL licensor nor licensee (nor AGPL for that matter), but having been party to the legal conversations at large companies I understand why BSL is generally considered to be more acceptable than AGPL. BSL may or may not be the right license for this software, I have no opinion, but businesses don't care about arguments from ideological purity. If BSL satisfies their requirements better than AGPL, and anecdotally all the evidence seems to support this notion, then they will go with BSL software.
To put it another way, if OSI is serious about being in the conversation about the future of software licensing, they must address the reasons so many companies refuse to adopt AGPL software.
Transferring copyright to another company, without a contract with that company, for the most part is something a legal team isn't going to agree to. BSL only addresses one side's concerns, and it's not in your favor.
AGPL, when used for internal services that customers don't interact with, is generally not that difficult to get approved. How many companies are using mongodb, for instance?
If it was AGPL I'd have been a happy camper though.
No, there are plenty that still use permissive licenses.
GitLab uses MIT and a custom license for EE: https://docs.gitlab.com/ee/development/licensing.html
Deno uses an MIT license and has some secret sauce that is currently just in hosted services AFAIK: https://github.com/denoland/deno/blob/main/LICENSE.md
PlanetScale has hosted services and an open source tool called Vitess which is Apache licensed: https://planetscale.com/ https://github.com/vitessio/vitess
Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/
Was excited to see the project but now seeing it is not Open Source it means 1/10th of value
https://www.gnu.org/licenses/license-list.en.html
https://opensource.org/licenses/alphabetical
That's a big red flag.
They are explicitly not a charity. Isn’t it okay to be a for profit company that sells software with a closed copyrighted source?
If Red Hat could change the license, would they? Maybe. (Not now but you can’t say the leadership won’t make that decision during financial crisis).
That there's freeloaders that benefit from the work without having to expend labour is a side-effect, not the primary motivation.
Chosing to release software under a non-FLOSS, proprietary license is hardly innovative, and I'm afraid trying to frame it that way sounds like you want to have your FLOSS cake and eat it too.
I ask this because I'm unsure if AWS Redis has any modifications on top of the Redis software itself, which would affect the speed, or even make it a bit slower. For example I know MS Azure's version of Redis restricted certain commands, and from a quick search AWS does something similar: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
(edit: added "affect the speed" for clarity)
Where's the benchmark compared to memcached?
Several years ago there was memcachedb, which could flush stuff to disk. While this operation was expensive, it was also useful, because you could restart instances without being overwhelmed by missing keys (data).
For the latter: your application quickly grinds to a halt if you need to build your cache from ground up after some kind of crash. This is a deal-breaker for many.
static constexpr unsigned NUM_SLOTS = Policy::kSlotNum;
static constexpr unsigned BUCKET_CNT = Policy::kBucketNum;
static constexpr unsigned STASH_BUCKET_NUM = Policy::kStashBucketNum;
Why not reuse seastar framework?
Can you describe your distributed log thing? Is it like facebook-logdevice or apache-bookeeper?
With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++. My choice was using fibers - which provide more natural way of writing code.
I have not designed the distributed long thingy. Will do it in the next 2 months.
Out of curiosity, are you discovering any new bottlenecks to performance outside of the software, given Dragonfly is able to process far more qps than most systems? I imagine the network and disk I/O could become stressed, but also I wonder if it breaks any assumptions of cross-core performance, hypervisors, etc. I know that cloud offerings typically mean that you can attach ginormous disk IOPS and NICs, but surely there are limits.
I was mostly running on AWS. In terms of hardware, for small-packets loadtests, most systems are constrained on throughput, i.e. number of packets per second. Some instances saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond in throughput. The best systems network-wise are c6gn family types. They are also better than instances that other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.
In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem: With share nothing architecture, I use a lot of thread-local counters that I aggregate only when stats are pulled.
As a general note, I expect that Dragonfly will stay very performant with the tailwinds from recent hardware advancements. For example, c7g (Graviton 3) is much better than c6g and DF shows it.
See https://s3-docs.fd.io/vpp/22.06/developer/extras/vcl_ldprelo...
dragonfly is a linking of a library and a main file dfly_main.cc so without this file you will have the lib.
Redis is "Remote Dictionary Server". You gonna loose the remote part :)
I will continue working on DF. Primary/Secondary replication is my next milestone.
Looks awesome so far, though!
We plan to implement everything but your votes can affect the priority of the tasks.
We do provide atomicity guarantees for all operations like Redis! We use an algorithm from a 2014 paper - see our readme, we provide the link to the paper.
Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.
They don't fsync the wal on every write, it's probably done as group commit every x seconds or every x MB.
> I would expect stronger guarantees from databases.
You can configure it and get lower performance.
> Maybe it's inpractical to expect this in cloud environments...
For benchmarks they use raid0 on local disks(not EBS) in aws vps.
I think came to conclusion that it could be interesting as an independent (novel) store but not something that can implement Redis with its complicated multi-model API, transactions and blocking commands. I do not remember all the details though...
Basically, I worked in a cloud company in a team that provided a managed service for Redis and Memcached. I witnessed lots of problems that our customers experienced due to scale problems of Redis. I knew that these problems are solveable but only if the whole system would be redesigned from scratch. At some point I decided to challenge the status quo, so I left the company and..and here we are.
One thing in common - we both thought that cache-based heuristics can be largely improved compared to memcached/redis implementations. We did it differently though. I think our cache design has academic novelty - I will write a separate post about it.
Can we see such disparity in benchmark even if we run Ncore instances of redis in parallel?