Lua: A Guide for Redis Users | Hacker News Reader

Lua: A Guide for Redis Users

redisgreen.net

123 pointsergot28 comments

This is a really good article, I referred to it a lot when working with Lua and Redis. Like a lot of people, I really admire antirez. The pragmatism he displayed choosing Lua was exceptional imo:

>> I think that for what we need Lua beats everybody else hands down. The language is not one that I particularly like, compared to Ruby for instance, but who cares? We are programmers and can code a short script in any language we want, but the point is, Lua is a wonderful implementation. Easy to embed, without even a configure script, like Redis! And FAST.[1]

[1] http://oldblog.antirez.com/post/redis-and-scripting.html

Honestly, it’s a great article. That said, there’s nothing wrong with nils in tables in Lua if you know what you’re doing. Yes, they can introduce weird behaviors, (particularly with the # operator) but if you understand how next, pairs, ipairs, and # work, which are required if you want predictable table iteration behavior in Lua, then this shouldn’t be an issue.

I think it’s dangerous to say that nils terminate tables if you’re not going to explain table iteration in Lua.

Edit: Here's a picture example showing how easily nils can cause confusion if you're using a #table based loop. https://i.imgur.com/qW0XoSs.png

for i,v in pairs(table) will go through an entire table.

for i,v in ipairs(table) will stop at a nil. It's meant for consistent behavior in tables with numerical indices.

The easiest way to have a value which "acts like null" but doesn't mess with table indexing is to use `false`, which is the only Lua value other than nil which is falsey (0 and "" are truthy).

Yeah, I wish Lua hadn't taken undefined and nil to be the same thing given the way tables work in it. Most of the time it doesn't matter, thankfully. Sometimes you bump into it.

A sigil value can work too if you absolutely need it. ex:

  undef = {}

  function isundef(val)
    return undef == val
  end

The sigil value is truthy and makes your code more convoluted, though. ex:

  if not val or isundef(val) then -- ...

So, :/. I do love the language though.

If you want, you can add t.exists[key]=true, and check it if you really want to know if key exists despite it's value is nil.

    function exists(t, k)
        return t[k] ~= nil or t.exists[k]
    end
    
    t = { 1, 2, nil, 4, exists={ [3]=true } }
    i = math.random(4)
    if exists(t, i) then
        ...

That's pretty close to Perl's semantics, where undef is inoperable under 'use strict' but arrays/hashes can have them explicit. Also can move exists out of t easily with weak table.

    local exists_t = setmetatable({ }, { __mode='k' })

    function set_exists(t, k)
        exists_t[t] = exists_t[t] or { }
        exists_t[t][k] = true
    end
    
    function exists(t, k)
        return t[k] ~= nil or exists_t[t][k]
    end

That said, I really like math-like attitude of definitions in Lua. No bs like 'bloated for your convenience', you just use your logic skills to program.

The key phrase to keep in mind in Lua is "sequence". The # operator on a table, and the ipairs function, are meant to be used with sequences. Per the Lua manual

  We use the term sequence to denote a table where the set of
  all positive numeric keys is equal to {1..n} for some non
  negative integer n, which is called the length of the
  sequence (see §3.4.7).

If a table contains numeric keys with any gaps (i.e. nil values between sequential numeric indices), then it's not a sequence. Because Lua uses a binary search to find the end of the sequence, it might select any nil as the one ending the sequence.

Also, another thing to remember is that a Lua table can have both a sequence and non-numerical (e.g. string) keys. So

  local t = { 1, 2, 3, n = 3 }

is a valid sequence. Using n to store the length of the "array" part is a common Lua idiom when you want array behavior but cannot guarantee a valid sequence. For example, table.pack creates a table out of a variable argument list; but because any of the arguments might be nil, it sets n as the length of the argument list. Of course, you have to explicitly make use of n in those cases.

Lots of people complain about this aspect of Lua. But Lua is very minimalist and there aren't any obvious alternatives. Lua could try to keep track of the maximum numeric index, but that can have poor asymptotic behavior unless Lua used a binary tree for tables instead of arrays and hashes. Lua could implement a type-safe array object, but that violates a core design guideline of Lua, which is that it provides the table as the only primitive for compound data structures. And in any event, it's trivial to implement your own array implementation using metamethods to maintain the invariants of a table while providing transparent support for # and ipairs.

This is the best line from the article:

> The best scripts simply extend the existing Redis vocabulary of small atomic data operations with the smallest bit of logic necessary.

Having used quite a bit of Lua (in Redis) I whole heartedly agree.

Yeah that’s generally how Lua should be used.

I use redis + lua quite a bit in my home-grown analytics real time tool. My biggest fear is when do I know I am overusing lua ? Is there a quantitative measure for "smallest bit" of logic ? I fear I have been given a very long rope with Lua + Redis and I may be unknowingly about to hang myself.

I don't understand what your concern is.

Lua is a relatively modern script language. It has all the tools you need to create modular programs in whatever style you like. Imperative is easy, OO style nearly so (there are plenty of OO implementations to choose from), and there is even some support for functional programming.

The PUC-Rio (main) implementation has a fast bytecode interpreter, and LuaJIT is wicked fast compared to just about anything else.

People don't tend to write really huge applications in Lua, but that is more due to the lack of libraries rather than anything else.

> People don't tend to write really huge applications in Lua, but that is more due to the lack of libraries rather than anything else.

We used it to write almost all the game-logic on a few titles. Ran great on all sorts of platforms(this was before LuaJIT). Coroutines are awesome for AI, lets you bundle in all sorts of implicit state easily and concisely.

> Is there a quantitative measure for "smallest bit" of logic ?

I don't see that as a valid concern anymore. The main concerns are 1) size of script you can comprehend, debug, and maintain and 2) do the most with the fewest network/redis calls. If you're 200 line lua script replaces 5-10 individual redis calls, that's surely a win simply from the perspective of network latency and data locality.

I really like the way this article introduces the concepts of integrating Lua into a storage system like Redis.

Shameless plug: the integration of Lua into Redis served as inspiration for our recent integration of Lua into Ceph/RADOS [1] for building custom object interfaces. I'll taking a cue from this article when we update our out-of-date documentation on the feature [2].

[1]: https://github.com/ceph/ceph/pull/7338

[2]: http://noahdesu.github.io/2014/01/22/dynamic-rados-object-in...

Are redis scripts still recommended now that redis modules are a thing? One of the pain points of scripting is the lack of test coverage.

If I was just trying to group 3 operations together into a "transaction", a script would seem like the way to go. But yeah, as soon as my "script" got at all long/complicated, I'd take a long look at making it into a module.

Or you could use a Redis transaction for a transaction. :-)

http://redis.io/topics/transactions

There is no reason why modules would be easier to test than scripts. It is probably the opposite.

Modules are great when you want to use Redis as a server for something it does not support, but I would still use scripting for most things.

I keep a bunch of small scripts for common tasks like deleting or aggregating keys maching certain criteria.

I wish redis offered a mechanism to save and call these scripts from the server, akin to the usual database functions and stored procedures.

I wonder if this was a conscious decision or something that's in the pipeline eventually.

What about this?

http://redis.io/commands/script-load http://redis.io/commands/evalsha

EDIT:

I realize this was mentioned in the article, just wondering why it doesn't fit your needs.

I believe the ask is to have any scripts loaded to persist through restarts of Redis. As is, you have to reload them on each init of Redis. As such, the onus is on applications to catch evalsha errors and load the missing scripts, in the case of Redis being restarted.

We built the same mechanism using HashMap and `SCRIPT LOAD` on restart/deployment to make a mapping table between function name and its SHA. Any client want to call the function just need to fetch the table and using the according SHA via `EVALSHA` command.

Is there a way to automatically load a script or a directory of scripts when the redis-server starts?

I don't think that's the recommended way of doing things. Redis will cache compiled Lua scripts internally so after the first EVAL you can call EVALSHA and pass the scripts hash. But you're client code should always be ready for the case that maybe the script is not there any more. In which case you call eval again. Automated tools that upload all the scripts would have no way to say "compile and cache this but don't run it" so you'd need your scripts to handle not doing anything. The usual pattern, then, is to simply send EVALSHA always and if it fails then call EVAL. With small numbers of scripts and larges numbers of operations this will be more efficient and doesn't require any co-ordination with the Redis server when scripts change or servers restart.

http://redis.io/commands/script-load

While I agree that the caller should be ready for a script not existing, one could build an automated tool to load scripts when starting redis using the SCRIPT LOAD command.

Ah I missed that capability thanks for the correction

Tangeant: it's so obnoxious that every code snippet from Github has to have such a signature. All I could read was "hosted with <3 by GitHub"

That's probably because it's not what gist is made for. Embedding a gist instead of adding a fenced block in your markdown or whatever blog engine you're using is kinda silly for a couple of lines of code.

https://gist.github.com/bpo/044fed4c53fc61a1a0a0

Author here. I completely agree. Embedded gists didn't have this annoyance when I wrote the article. I figured GitHub could be relied on to not change things up too much, but that's what I get for depending on an external service...

Reply on news.ycombinator.com