Redis scripts do not expire keys atomically | Hacker News Reader

Redis scripts do not expire keys atomically

ably.com

67 pointsstichers36 comments

Redis doesn't guarantee that expiry happens exactly at expiry time ([1] gives up to a 1 millisecond delay), so if your system assumes that all related keys are removed at exactly the same time it is still broken (although the race window is now tighter).

Maybe you could store the related keys in a hash or something?

I'm also guilty of trying to use features in redis in ways they weren't really designed for - it's such a fantastic bit of technology that a lot of the time you can still get away with it but it's not redis's fault when it doesn't work!

1: https://redis.io/commands/expire

Right. Relying on atomic and absolutely reliable expiration is the problem anyway. If it's really that important then it needs to run in a transaction that manually sweeps the keys.

yep - part of our fix was ordering the expires such that the invariants are maintained even if the expires arn't exactly at the same time (we are assuming expires are at least ordered)

I'm pretty sure they aren't ordered. Redis active key expiry is a random process: https://redis.io/commands/expire#how-redis-expires-keys. TL;DR: Redis will randomly sample the keyspace and delete any expired keys it finds in the sample. If a large enough percentage of the sample was expired, it will repeat the process. Since the sample is random, it cannot be ordered.

There is also a "passive" key expiry where any read to a key will check the expiry time and delete the key if the current time is larger than the expiry time.

I’m not aware of the docs making any guarantees that expires are ordered either, just that at some point after the expiry time the keys will be expired.

sorry i should have explained. by ordered i meant after expiring a key its TTL will be set, which will never be a time before a previous expire. when it looks up the key, if its TTL has been reached it will not be returned (which we checked in the redis source). we weren't concerned about then the keys are actually removed

why are you assuming something like this? It's not guaranteed in the documentation, did you at least check that the implementation actually does it?

>we are assuming expires are at least ordered

Why are you assuming that? And if true, why do you think that assumption will hold in the future????? Redis could always tweak or change the behavior.

Redis expires can't be ordered, unfortunately. [0]

One hack I can think of is, accessing the keys. If the key is expired, then Redis won't provide it and you could know it is expired.

Another way is to club all of the relative keys in a hash [1]

[0] - https://redis.io/commands/expire#how-redis-expires-keys

[1] - https://redis.io/commands/hset

yep - as i mentioned in a comment below - we arn't concerned about when the key is actually deleted - only that its not returned when looked up if its TTL is hit

sorry should have been more clear

no need to say sorry! Thanks for responding, and the post was fun to read.

>The integrity of a set of related keys requires that either all keys exist, or none exist.

Using a Key-Value DB to store relational data eh - that never leads to problems.

Their solution isn't any better because the API does not guarantee a particular order or atomicity of key expiration - so great that it works for you now[1], but next version may change the behavior.

[1] Does it work though? Are you really really sure?

>The integrity of a set of related keys requires that either all keys exist, or none exist

Sorry it wasent that clear that part of our fix was such that it no longer matters if all keys exist or none - we reordered the expires such that the invariants still hold even if all the keys dont still exist My understanding of expiry is - its not guaranteed when keys are expired if they are not accessed - but if a looked up keys TTL is hit it will not be returned - which is all we cared about

It should be noted that an EXPIREAT triggers a deletion if the timestamp being set is in the past and could lead to an increase in memory and disk pressure during script execution. Worth checking if the behavior there differs in any meaningful way since y'all made the change. EXPIRE usually just marks a key and relies on either subsequent operations to the same key or garbage collection to actually excise the entry and do the subsequent bookkeeping, I believe.

It's been a while since I had to deal with this kind of problem though, so please tell me if things are different now.

It seems the article's implication is that they are relying on redis TTL and expiry to guarantee all keys exists or no keys exist, at all times, for a given key set.

I never got the impression that Redis gave guarantees on when expired keys were deleted, just that they would be eventually.

If the issue is that you want a consistent all or nothing, why not just use a single key to make that determination?

we reordered our keys such that they can expire at different times - also redis checks a keys TTL before returning - and wont return if its expired. i think it will 'eventually' delete if its not accessed, but we only cared about keys being returned, not when its actually deleted

Thanks for clarifying. I wasn't aware of the fact that redis removes expired keys even if they aren't deleted. I've always thought of expiry as more of a resource management annotation. Interesting way to use it.

I think the solutions to this problem are:

- If the keys are strings, just use a single hashmap and you're done.

- If the keys have different types, then you need a Redis module that implements something like `MPEXPIREAT time key1 key2 ...` and then you need to either be able to handle the case where you successfully retrieve key1 but not key2 (because they expired half-way through) or you use a lua script to execute the multi-key access atomically. That said, without the lua script the whole point of having the keys expire at the same time makes less sense, IMO.

Hey, I'm the author of this blog post. Ask me anything!

Would Redis Keyspace Notifications [1] help in this case? Like a script that runs when a specific key expires, such that all other related keys can be deleted too.

[1] https://redis.io/topics/notifications

thanks for pointing that out - will take a look :)

these notifications aren't guaranteed to fire at the expire time, wouldn't that be same as current problem?

I don't want to sound snarky, but since you mentioned that Redis does not document if/that "multiple key expiry is guaranteed to occur at the same time if keys have the same EXPIREAT setting", I am tempted to ask - why did you not check the implementation you're using? That's the unique selling point of using FOSS - you don't have rely on documentation or observational guesswork - you can wade in and really see for yourself! Redis' source code is especially readable in my opinion (thanks again, antirez, for creating this little load-bearing gem of a data structure server), so I would very much encourage you to try, even if C is not your particular forte :)

yep good point :) - in the end it didnt matter much as we could maintain our invariants if we just reordered the expires

Not who you're asking but... why have documentation at all? So that we don't have to read source code.

You can’t cover every usage possibility in docs… having the code available allows you to check expected behaviour

You also can’t tell what is supported behavior and what is implementation coincidence from the code…

Because documentation is what is needs to be: A simplified and necessarily incomplete description of program behavior. It's a trade-off, since a complete and exhaustive description of all intricacies of potential program behavior would be at least as complex as the definition (= its source code) of said program.

If you happen to have a question that ends up on the wrong side of that trade-off between documentation's completeness and accessibility, you will have to descend into the depths that lie beneath. I believe the redis documentation actually strikes a rather OK balance in that regard.

Good docs generally make guarantees and pitfalls clear.

Of course having documentation is better but that's not a reason not to read the source code if the documentation isn't enough.

Because good docs are like a good search engine: high hit rate for what you want most of the time.

The edge cases and quirky impl detail questions are what src is for.

I would argue that outlining expected behaviour in edge cases is one of the most important topics for documentation to cover.

> The edge cases and quirky impl detail questions are what src is for.

Literally the most important thing you can document.

That's a great suggestion, and OP should do this, but even if OP verifies behaviour today through code inspection, it's possible a future release could change it. The problem is that Redis doesn't want to make that guarantee so you can't rely on it.

I ran into a similar issue and used a hashset of all the related keys with ttls on the individual keys. the hashset would expire making the keys unreachable, and the TTls basically garbages collects the individual keys. anything that required to transact with the set would do so atomically by being wrapped in a lua script.

Is there a reason that the DEL command wouldn’t work? Is there some semantic difference with expire?

Reply on news.ycombinator.com