Transactions in Redis | Hacker News Reader

Transactions in Redis

dr-josiah.com

60 pointsDrJosiah7 comments

Interesting concept! Just curious if you have any reports on how this affects the overall execution of a script, as well as the memory consumption and CPU usage? (Not in a place the moment to test it myself :P)

The existing implementation is a proof of concept. It works as advertised, but any key that is written to as part of script execution gets dumped using the same internals that the DUMP command uses.

For a handful of small keys, where EVALTXN/EVALSHATXN calls are coming one at a time from your clients, the performance difference is relatively minor - on the order of a few percent.

But for large keys, or large numbers of modified keys, because every key is being dumped before being written to the first time (and because the current implementation uses a singly-linked list to manage the rollback buffer, leading to a linear search for whether a key has been changed), the slowdown is substantial, and there is no upper bound on the slowdown.

But how can there be no upper bound on the slowdown? How big are your keys, and how many are you changing? If you're dumping 5x string keys all of under a few hundred bytes as part of each script execution, Redis is not going to have a problem. But if you are dumping 1000x 1M element ZSETs, of course it's going to be slow!

The point of the patch is to get people (like you) asking questions and hopefully wanting to help. There isn't any good reason for Redis to not have transactions, aside from someone not taking the time to do it. The current implementation is crap. But the crap implementation is substantially better than not having it.

The next step? Teaching data structures how to offer the ability to reverse operations on themselves. Or fetching versions from caught-up slaves. Or (now that fork() performance is good due to no transparent hugepages) pull the data from a fork()ed child. Or write changed structures to disk in a secondary thread in the background, force structure syncs on write (if there are changes), and pull them from disk on rollback (as necessary).

There are several different ways that this can be improved substantially, and the only thing stopping it right now is time and effort. I will be doing my best to work on it further (I had been sitting on the original version of the patch for 2 months because I didn't have time to work on it), but that doesn't mean that Redis has to wait for me. I know there are people out there, Salvatore included, who can help take the idea and turn it into something that everyone uses and expects to be available, because the current state of affairs is terribly unfortunate and a recipe for data loss/corruption (which has happened to me).

I'd save the commands to a buffer and apply them at the successful termination of the script. See "batched staging", trying to reverse a data structure already mutated will be problematic. This is actually how one should write lua scripts in Redis anyway.

I'd agree to this whole heartedly. At first glance (if I'm understanding your implementation correctly), it appears like you're already going to use up the same amount of space a buffer would anyway. In this case you're just tasking the LUA best practice, but making it a feature within Redis itself.

Regardless I think you're really onto something that people want. There's going to be a trade off one way or another, but the Redis community is all about speed, so the faster and more efficient you can do it, the better (:

I agree on the speed side of things... but see my reply to the other comment. TL;DR; Buffer or not, mutations are necessary to be visible as part of script execution, which precludes "only write at the end on success".

The problem is: when you are mutating the structures as part of the script, and you are accessing the results of those mutations, you can't just "apply them at the successful termination of the script". The successful termination requires that the mutations be visible - aka the mutations already applied.

In a relational database this isn't normally a big deal, because the structures themselves (rows and indexes) are already versioned, and most rows are of a relatively-small size. So rows typically work well for intent/write-ahead logs, old/new version logs, etc. But with Redis, operations aren't just on a "row", it's on a structure, and that structure can be gigabytes in size (I had a 30 gigabyte ZSET about 4 1/2 years ago, that was no fun during snapshotting, let me tell you).

Reply on news.ycombinator.com