reconditerose

LuaJIT is mostly compatible with 5.1, our goal is to make it pluggable so you can run with either 5.4 or 5.1.

Seems similar in impact to https://nvd.nist.gov/vuln/detail/cve-2021-32626, I wonder why this has a CVE 10.

This code also looks generally fixed in Lua5.4, https://github.com/lua/lua/blame/9ea06e61f20ae34974226074fc6.... Valkey and Redis really need to move to Lua that isn't so old.

Google is well known for having banned AGPL (https://opensource.google/documentation/reference/thirdparty...).

The tl;dr is it's just a lockless hashmap attached to a TCP server with a log. Simple Get/Set operations are highly optimized, so with high batching they are able to efficiently fetch a lot of data efficiently. The architectures scales very well when you add threads and data access that is uniform.

It struggles a bit on certain types of workloads like hot keys, think heavy hitting a single sorted set. It's a cool architecture.

> So, you are practically saying you were using, at Amazon, all the BSD code we provided, provide an important part of the code to us? You see how broken such model was? At least stop defending it.

I'm not defending it, I'm trying to fix it. I want Amazon to contribute back. That's what I spend most of my time doing, but I can't just sit in a meeting and tell people we should give away code. It takes time to convince people that we should collaborate on the core and just compete on what we want to differentiate on. It takes time to convince people that building open-source in a vendor neutral space makes software that is better for everyone.

I hope that makes sense.

My guess is they are making it up. AWS has no public information, but there are some high profile customers that have migrated https://aws.amazon.com/elasticache/customers/.

Do you have data to back up your claims? I see a lot of customer claims for Valkey here, https://aws.amazon.com/elasticache/customers/. Neither of the AWS or GCP offerings are in preview.

The Valkey implementation of multi-threading is fundamentally different than what existed in Redis. The history dates back to work done in ElastiCache that was released as "Enhanced IO", https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-el.... The version released in Redis could only do about 350k RPS because of poor memory locality of operations, the inability to do command processing while handling I/O, and the inability to offload much of the TCP read path. The new version in Valkey can achieve 1.2M RPS.

"They made certain improvements later", should be "we threw away the old implementation and built a better one."

It's not just a fork, there have been two releases on Valkey that improved performance and memory efficiency. There is a lie that Redis likes to spread that only their own employees were working on the core engine at the time of the fork, but most of the engineers on Valkey came directly from having worked on Redis OSS. A recent example is we modernized the hash table a bit: https://valkey.io/blog/new-hash-table/.

Yeah, there has been a lot of stuff like performance [1] and efficiency improvements [2]. A lot of the contributors, that didn't work for Redis labs but worked on Redis OSS before the fork, moved to Valkey and they continued to contribute.

[1] https://valkey.io/blog/unlock-one-million-rps-part2/ [2] https://valkey.io/blog/new-hash-table/

Everything we do is open-source, so you're free to take it and incorporate it back into Redis. (Although I'm more into rewriting the data structures to more efficient on modern CPU hardware). Keep some comments out in the open like your discussion about vector sets to keep the conversation going. For the record, I really don't like the vector set idea, but I'm not one to dish out hot takes on hacker news.

As one of those 12 other people, number of commits is not a good metric in this context. When you have a single maintainer, like antirez was for a long time, he could commit 10+ commits for a single small feature but I needed to get someone else to approve my commit before merging.

I know this doesn't mean much, but I spent a lot of time outside of work helping to maintain the Redis codebase. I attended our meetings and responded to issues on vacation because I cared about the community, it was more than just a paycheck for me. I don't regret doing any of that.

Valkey maintainer here. ¯\_(ツ)_/¯

I worked with antirez a bit before he left the project in 2020, and I still look fondly back on that time since he was a wonderful person and he helped me learn a lot. I wish he had considered coming to work with Valkey instead of Redis, but maybe we'll find some some way to work together that benefits both communities.

The only public record I've seen him talking about it was in https://github.com/valkey-io/valkey/issues/544, where he mentioned having sold the copyright.

There is a lot of adoption of Valkey. I know it's fun to hate on AWS, but there is a wider shift than just them.

https://cloud.google.com/blog/products/databases/announcing-... https://upcloud.com/blog/now-supporting-valkey https://aiven.io/blog/introducing-aiven-for-valkey https://www.instaclustr.com/blog/valkey-now-available/ https://elest.io/open-source/valkey

We're working on it, you can follow the progress here: https://github.com/valkey-io/valkey/issues/640.

It was always, and still is, a license issue. Redis stack had a proprietary license and now Redis has a proprietary license.

You might be able to get near 100% of Redis performance without a lot of tuning. Redis has a lot of overhead to make it a production ready system like performance counters, safety checking, and conditionals for features like replication/clustering/etc which slows it down.

Hell, you might even be able to use a concurrent hash map implementation and show multi-threaded performance much better than Redis :).

> Fair point. Was that foreseeable in 2009? A lot of the actions we take now are based on what's reasonably foreseeable in the future. I honestly don't recall.

The company now known as Redis didn't acquire the redis trademark until 2018, and didn't change their name to include redis until after that. That is probably the better timeframe to think about.

The other is the company that bought the trademark https://github.com/valkey-io/valkey/issues/54.

* Google added support to Valkey: https://cloud.google.com/memorystore

* AWS says they are moving: https://aws.amazon.com/blogs/opensource/why-aws-supports-val...

* Aiven added Valkey: https://aiven.io/blog/introducing-aiven-for-valkey

* Instaclustr mentioned moving: https://www.instaclustr.com/blog/redis-to-valkey/

* Oracle indicated support: https://blogs.oracle.com/cloud-infrastructure/post/oracle-su...

Azure is the main one sticking with Redis: https://azure.microsoft.com/de-de/blog/redis-license-update-...

I might edit this if I remember some more.

I'm from the project.

There shouldn't be any caveats of replacing Redis 7.2 and early to Valkey 8.0. I've talked with a few folks who have migrated and none so far have hit any issues, one even migrated from Redis 2.6.

Yeah, right now the valkey project is trying to keep full compatibility with the lass oss version of redis but improving where we can. There are plenty of concurrent hash maps attached to a tcp server that can out perform it on raw throughput, but they're usually missing a lot of features. One day valkey might be fully multithreaded, but not anytime soon. (Unless someone has a good idea for it)

Haha, we definitely used profiling tools. :) Perf is great and definitely the first tool to use. I know vtune has been used as well.

The biggest per core "efficiency" improvement will come from RDMA, since IO tends to be the largest bottleneck in Redis (and now Valkey). However, if you are running with deep pipelines (sending multiple commands as a batch without waiting for responses from each commands) than the benefits of RDMA are pretty limited since the bottleneck is on command processing. Multithreading doesn't help much either.

One of the strategies that is being used in the multi-threading is using CPU memory prefetching to pull memory closing to the CPU so that we aren't stalling on fetching data from main memory while executing commands. We still want to try to apply these techniques without multi-threading to improve the efficiency of single or double core installations.

This is basically the original source of the claim, https://valkey.io/blog/unlock-one-million-rps/. We are working on a followup that goes more into depth about how to setup the benchmark so that is easier to reproduce.

FWIW, Redis was previously mostly developed by the community but the trademark was acquired by a VC funded tech company. AWS, and other cloud providers like Tencent, were contributing to Redis and they went ahead and changed the license anyways. You can read more about it here, https://lwn.net/Articles/966631/.

Valkey (an open-source alternative to Redis) has it's first release candidate for 8.0. A sample of some of the features:

* A new I/O threading architecture that allows dynamically adding and removing threads and increasing the max throughput by almost 3x in some cases. * A new hash table layout for cluster mode that reduces per-key overhead ~15%. * Intelligently embedding keys into the main hash table to further reduce overhead ~8% and improve memory locality: * Share slot-migration state with replicas to make clusters more stable during failure modes. * Send the full-sync over a dedicated channel to improve bootstrapping new nodes into a cluster (up to 50% faster and with less copy on write). * New opt-in cluster metrics to identify hot slots in cluster mode. This should make it easy to plan rebalancing operations.

I haven't looked at the topic since we created the fork, and I'm surprised people weren't still arguing about it. There were a lot of vary passionate people in the beginning, but very few of them actually contributed and most of them left pretty quickly.