Lua vs Ruby benchmarks (similarly "high-level" languages, insanely different performance) | Hacker News Reader

Lua vs Ruby benchmarks (similarly "high-level" languages, insanely different performance)

shootout.alioth.debian.org

39 pointscritic72 comments

LuaJIT is not the standard mainline Lua interpreter that most people think of as "Lua the language". It's still a fairly young, experimental project and you wouldn't want to write production code in it. However, the standard Lua distribution is still much faster than Ruby.

But even though both are dynamic interpreted languages, it's still apples to oranges - they have very different design goals.

Still, I think it's a copout to say performance doesn't matter in any general-purpose language the way the core Ruby team seems to. Micro-optimization is a waste of time, but efficiency and intelligent implementation choices are necessary.

It also only runs on i386.

In practice (not micro-benchmarks), I've found LuaJIT to be at least twice as fast as the standard version, and in some cases three or better.

Without some discussion of why Lua is faster, this isn't very interesting. Lua is known for being small and fast, so this benchmark shouldn't be news to most people here.

SBCL is faster still, averaging about half the speed of GCC and about the same as Java.

SBCL is not as "high-level", especially when you are aiming for speed. Yossi already wrote about this http://www.yosefk.com/blog/the-high-level-cpu-challenge.html , so there is no need for me to do it:

    If you use Lisp’s static annotation 
    system, your code becomes uglier than 
    Java, and much less safe (I don’t think 
    Lisp does static checking of parameter 
    types, it just goes ahead and passes you 
    an object and lets you think it’s an integer).

Additionally, Lua only has "one data structure type" - that's also "higher-level" than what you have to do in Lisp, deciding among singly-linked lists, arrays, hash tables, binary trees, lexical trees, etc.

So while SBCL has its place, there's something about Lua you have to admire, don't you think?

SBCL actually does a lot of inferred type checking when type declarations are present.

The first place to look is probably "The Implementation of Lua 5.0" (http://www.tecgraf.puc-rio.br/~lhf/ftp/doc/sblp2005.pdf).

Short paper, but very interesting.

The LPEG paper (http://www.inf.puc-rio.br/~roberto/docs/peg.pdf) is great, too.

The "History of Lua" paper written a few years ago for a presentation at HOPL is also great reading and gives good insight into why Lua is the way it is.

http://www.tecgraf.puc-rio.br/~lhf/ftp/doc/hopl.pdf

The important difference is memory. It's generally possible to trade space for speed in a JIT and accelerate anything. SBCL is lisp with a JIT. It's as fast as Java - it's as fat as Java (to within 5x either way). Lua is both small and fast, and is more or less on its own as an interpreted language in that category. (Forth is smaller and faster, but forth is about as low level as macro assembler.)

I don't know Lua, but this is the first time I've heard Lua and Ruby equated in terms of expressibility/flexibility (i.e. "high-level"?). Is that a particularly relevant comparison?

Me too. It motivated me to spend the last 15 minutes looking into Lua.

Similarities: Block/lambda support and syntax, optional parentheses, control statements as block, prefix, or postfix.

OTOH, "how come Lua is not as widespread as say, Python ? I'd say the answer is this: there is no real standard library, and as opposed to Python, Lua doesn't come with batteries included." -- http://www.ivy.fr/blog/index.php/2008/03/17/83-kahlua-lua-on...

Very true, Lua is optimized for being very small, fast, and portable. It optimizes for embedding, which means you cannot make assumptions about what you are allowed to do, so it errs on the side of very little.

On the other hand, interfacing to C is the nicest of anything I have done, by a long shot, so the lack of batteries isn't that painful in practice. It fills a different niche than python or ruby, in its, it shines.

Lua presumes you've already got your own batteries, thank you, and you don't want those crummy off-brand ones that always seem to come "included".

Thus, I feel a Lua - to - Ruby/Python comparison is sort of erroneous. Use Lua when you already have all your tools. Use Ruby/Python when you don't have any tools, and you want cool whizbang higher-level stuff.

Honestly the benchmark is wholly unsurprising to me; Lua is a much more minimal language. If you need another language to lump Lua with, lump it with Javascript sooner than you would Ruby or Python.

However, as a longtime user of Lua, I would still greatly appreciate a default set of "batteries-included" libraries as an option, like what you'll find in Python, so that it could/would be that much faster to write simple applications in pure Lua, without needing to deal with compiling/bridging to C/C++ libraries...

As a longtime Lua user, you're probably aware of this, but I'm also writing for the other people reading the thread... :)

Look at LuaRocks (http://www.luarocks.org/). It's an attempt at a CPAN-like package collection. The stdlib and bitlib libraries, in particular, are pretty necessary. (Also, http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/ has some others.)

Lua errs on the side of only putting what can be implemented fully portably in ANSI C in its standard libary, which makes it a bit spare.

Yeah, actually. Lua isn't as clean as Ruby (or Python, or your_favorite_language), but it has the advanced features you'd expect from a powerful scripting language: closures, coroutines, etc. Rather, Lua is like a really experienced prostitute. Not necessarily beautiful, but can do all the tricks and can get the job done efficiently and inexpensively.

To get started with Lua, I'd recommend PIL: http://www.lua.org/pil/

Are you serious? Lua has an extremely clean and orthogonal core. It's got serious CS mojo. The "batteries not included" bit is probably the greatest detraction to Lua as a general-purpose scripting language.

Certainly. I didn't mean to deride anyone's work with the comparison. I meant, some of the implementation decisions seem arbitrary, and seem like they were made "for the sake of being different".

For example, Lua's use of two dashes for line comments, and --[[ and --]] for block quotes. That lets you remove the first --[[ in a block quote without causing the program to break, because the "closing" block quote is commented out at that point (because --]] is a comment at that point). But really, that's so different from other languages that it seems like a liability rather than an asset. That's what I meant by "not necessarily beautiful". It is definitely beautiful in certain other aspects, though.

Also, if I were writing a scripting language, I'd love for it to be "like a prostitute". Even if nobody respected it, everyone would be using it.

> Even if nobody respected it, everyone would be using it.

By that metric, PHP is the scripting language for you. ;)

The double hyphen comment is used elsewhere. SQL comes to mind.

In that vein, don't array indexes start with 1, instead of 0? Erlang does that too, and it's a bit weird, but not a show stopper.

Right, exactly.

Lua doesn't even really have arrays per se, but rather "tables," which are a light, flexible data structure somewhat similar to Javascript's objects. You're free to index tables starting with 0 if you really want to, but the convention in Lua is to use 1.

This is slightly less true than it used to be. The table length "#" operator returns the index of the largest integer key continuously reachable from the index at 1. (i.e. there can be no integer keys with nil values in between). This, as well as a handful of standard library operators, rely on the index-from-1 convention.

Tables in Lua can be thought of as discrete mappings of any value as key, to any value as value. The reference implementation happens to have the property that if you index from 1 upwards by increments of 1 that all of those values are kept in a single C array which is then subscripted in the usual fashion. It's a handy thing to know when you decide how to lay out your data. The "dictionary" portion of the table only springs into existence when something is actually inserted into it.

Actually if I understand correctly, an empty Lua table is basically just a pointer to a minimal structure in memory, so it can be used for identity comparisons and so on where only uniqueness is necessary, rather than a full array+dict functionality with the associated memory overhead.

Lua tables are extremely flexible and powerful. Especially when combined with the "metatable" concept.

The table length "#" operator returns the index of the largest integer key continuously reachable from the index at 1. (i.e. there can be no integer keys with nil values in between). This, as well as a handful of standard library operators, rely on the index-from-1 convention.

That does not always hold true. The '#' operator is weird. Here is an example:

  Lua 5.1.3  Copyright (C) 1994-2008 Lua.org, PUC-Rio
  > x = {1,2,3}
  > x[1] = nil

ipairs() stops at the first nil, as expected:

  > for i,v in ipairs(x) do print(i,v) end

However #x is inaccurate now:

  > print(#x)
  3

Luckily, we can always see all the values with pairs():

  > for k,v in pairs(x) do print(k,v) end
  2     2
  3     3

And watch what happens when we add a single non-integer key to the array:

  > x.a = 10
  > print(#x)
  1
  >

Looking at http://www.lua.org/source/5.1/ltable.c.html#luaH_getn I think the problem lies in the assumption in luaH_getn that if the table doesn't have an 'hash' part then it also doesn't have any gaps.

   /*
   ** Try to find a boundary in table `t'. A `boundary' is an integer index
   ** such that t[i] is non-nil and t[i+1] is nil (and 0 if t[1] is nil).
   */
   int luaH_getn (Table *t) {
     unsigned int j = t->sizearray;
     if (j > 0 && ttisnil(&t->array[j - 1])) {
       /* there is a boundary in the array part: (binary) search for it */
       unsigned int i = 0;
       while (j - i > 1) {
         unsigned int m = (i+j)/2;
         if (ttisnil(&t->array[m - 1])) j = m;
         else i = m;
       }
       return i;
     }
     /* else must find a boundary in hash part */
     else if (t->node == dummynode)  /* hash part is empty? */
       return j;  /* that is easy... */
     else return unbound_search(t, j);
   }

I did not get the same behavior as you when I tried this just now - rather, I had:

  > x.a = 10
  > print(#x)
  0
  >

Which makes more sense than 1. At the same time, 3 is also a valid value of #x, because it is the integer index of a value directly preceding an index with a nil value.

This behavior has been hashed over on the mailing list multiple times... the exact wording of the documentation allows for it but it's not exactly what people expect of the length operator.

The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n can be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t can be any of the indices that directly precedes a nil value (that is, it may consider _any_ such nil value as the end of the array). -- From the manual

My expectation is that there will be some changes to the handling of the # operator in 5.2 increasing its usefulness and allowing it to be overridden via metatables for user-defined types.

In any case, this is one of the few currently ugly corners of the language itself. Another would be the handling of destructuring varargs via the select(...) function, which can be really funky.

I did not get the same behavior as you when I tried this just now

What version of lua did you use? Perhaps my version's # is buggy in addition to being weird.

This is one of the things expected to be fixed in 5.1.4, IIRC.

# works with an array-style table (consecutive integer keys, starting from 1) and not hash-style tables. Once you're familiar with Lua, you rarely confuse them -- they tend to be used very differently. Using the same variable type is mostly an efficiency trick.

I agree that counting from 1 is kind of annoying, but part of their reasoning was that Lua is targeted as a scripting language for niches that can include people who don't really program. They tried to keep "weird" programming stuff from obfuscating the tiny amount necessary for writing config or data files in Lua. It has coroutines, lexical scope, metatables, etc. when you use it for real programming, but they don't get in the way when you don't need them.

For example, from the data file for my laptop's wifi script:

   device="ral0"
   sudo=true

   Network { key="home", nwid="barbelith", wpa="..." }
   Network { key="sparrows", nwid="The Sparrows" }
   Network { key="schulers", nwid="schuler-alpine" }

  Lua 5.1.3  Copyright (C) 1994-2008 Lua.org, PUC-Rio
  >

This is using the Lua console installed by the the (excellent) Lua for Windows package.

personally i think 0-indexing is weird. it makes sense as a pointer offset in lower-level languages. but in higher-level languages we have the opportunity to index the first element by the number 1, the second by the number 2, and so on instead of having to do that little -1 every time which is something that has to be learned

There is a short note by Dijkstra about this

http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EW...

that's really weird. was it written at a time when most programming was for scientific purposes? i don't know my comp history

if i parsed it correctly, he is saying that things should be 0-indexed because a particular mathematical notation corresponding to it is more sensical, but i don't see what mathematical notation has to do with programming notation, both of which are man-made, arbitrary, and for different purposes, such that reasoning about either with respect to the other is imo fallacious

he compares subscript ranges in a notation where 0-indexing appears to be better, but choosing a different notation (such as b in his examples) makes it look better for 1-indexing ( 0 < i <= N versus -1 < i <= N-1 )

Here is the important part:

    Extensive experience with Mesa has shown that 
    the use of the other three conventions has been
    a constant source of clumsiness and mistakes

0-based indexing is not something you can rigorously prove to be better.

One advantage of 0-based is that using array[mod(i, length)] works. It mostly seems to me like 0-based vs. 1-based is a relatively arbitrary decision, though. Each just leads to fencepost errors in different places.

if i read correctly, that's talking about the mathematical notations, not arrays or the indexing of such, of that programming language

and while we're considering sources of "clumsiness and mistakes," 0-indexing is an exceptional candidate, from my experiences helping a friend learn C++ and my own memories treading through those early days

i look at it from the perspective of usability

In C++ and C the iteration idiom almost always looks like:

  for (elementIndex = first-element; elementIndex < indexAfterLastElement; elementIndex++) {
  ... 
  }

That is, writing the equivalent of the mathematical notation:

  firstElementIndex <= elementIndex < indexAfterLastElement

Two advantages of that idiom:

The first one is that indexAfterLastElement - firstElementIndex equals the number of elements being scanned.

The second advantage (which is the most debatable, and personal) is that the initial boundary condition is trivial (it's the first element) and the end boundary condition is natural i.e. just look past the element you are looking at whether there is a next one. That's what Dijkstra is talking about when telling the example of the student who would not look past the end of the page.

I used the epithet natural because for most of the objects we interact with, the end is not part of the object, rather it is the transition in between the last element and nothingness.

Now that said (in too many words) if you compare 1-based arrays and 0-based arrays you have:

  for (i = 0; i < N; i++) {
  ...
  }

  for (i = 1; i < N + 1; i++) {
  ...
  }

Both would be acceptable. However 0 based arrays have the advantage of giving i two meanings:

  1. the index of the current element
  2. the number of elements scanned so far.

At the end of the iteration, i also conveniently contains the number of elements having been scanned. Which can be useful in algorithms where there is an extra condition which might let you leave the iteration early.

This to me is the most important part, zero-based indexing is more expressive.

These are some of the best arguments for 0-based indexing, and I agree that these are very useful properties. In my day-to-day programming though, the 1-based indexing has some (admittedly, less math-based) advantages too. For instance when I have an array my brain appreciates the simplicity of the "oh, this array has 10 elements. What's the last element? Number 10! Yay, that was easy" thought process.

In many cases I don't care about specifying boundary conditions at all; I will just use something along the lines of

  for i, v in ipairs(t) do something_interesting(i, v) end

or even just a map or fold or some other iteration technique where I'm not explicitly specifying boundary conditions. If I'm implementing a heap or some other data structure which uses calculations to interact with an array I will be double-checking my math anyway because I don't trust myself to get it right the first time, 0-based indexing or not ;)

The last element of an array is foo[-1] regardless of it's length. :-) And try implementing a circular buffer using an array of N elements with 1-based indexing and then 0-based. Sorry in advance for the loss of hair due to the former.

you mean like array[mod(i, length) + 1]

the [[ and ]] are used for literal/multiline strings, so they make sense in block quotes. and you don't have to end the block with --]]. ]] is fine, which is the only thing i've ever used since it would never have occurred to me, especially as an anemic commenter, to use that quoting trick

the [[ and ]] can be extended to nest strings/comments like so:

  --[==[
  function add( a, b )
      --[[
      return a + b
      ]]
      return a - b
  end
  ]==]

you can use [=[ ]=], [====[ ]====] etc

With the number of equals matching on the opening and closing comments.

Having done a lot of both, Lua feels more pythonic than rubyish in many small ways. Objects are just syntactic sugar where self is the first arg (though unlike python, lua has the syntactic sugar to make it prettier to look at), etc. Interestingly, typical OO Lua is prototype based, not class based -- in that way it is more javascriptish.

Most lua codebases I have seen tend to be written in what I call an "imperative functional" style, where the language isn't purely functional, but functions are the most common means of abstraction.

It has proper anonymous functions, lexical scoping, etc so you can write very rubyish code (such as moonunit http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/lu... and using it http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/lu... ).

A unit testing framework in 37 lines of code, not too shabby.

with color coded output ;-)

Yes.

In short, Lua's philosophy is to keep the core language as small as possible, but to include features that make it easy to add major language features as necessary. Adding an object system to the language is trivial, for example. (It doesn't have hygienic macros, though there are some side projects that attempt to add them. No experience with them, though...I'd just use Scheme.) Lua feels to me like a minimalistic, more internally consistent Python.

Personally, I would hav just done Lua vs Ruby instead of LuaJIT vs Ruby.

Results look almost the same ;-)

http://shootout.alioth.debian.org/u32/benchmark.php?test=all...

Lua is generally the fastest scripty language, ruby is generally the slowest. Ruby is much more general purpose though -- Lua is very optimized for embedding.

LuaJIT makes the Lua mandelbrot program run 95 times faster than Ruby's, whereas the stock Lua interpreter is only 17 times faster.

Better yet, LuaJIT is a drop-in replacement. There's no reason not to use it, as far as I know.

Well, LuaJIT only runs on x86, and only in 32-bit mode. Though, the work-in-progress LuaJIT 2 is supposed to be easily portable to ARM or other architectures.

I love the "only 17 times faster" bit ;-)

Try comparing ruby to c++!

mod_lua will be in the next stable apache release. http://lua-users.org/lists/lua-l/2008-12/msg00119.html

Who would care to implement Ruby in Lua?

While it would be an interesting project, the Ruby wouldn't necessarily be faster for it. I don't know Ruby well enough to say, but there are probably aspects in the language design itself that make it slower (as a trade-off for expressiveness). Lua is very fast, but that comes from aggressively simplifying the core of the language. Most things are ultimately built on only a few central constructs, and they have been very throughly optimized.

Which version of Ruby? 1.8 or 1.9? With my own projects Ruby 1.9 performs much better. Lua is well known for being small and fast, so I doubt Ruby 1.9 would beat it... Nevertheless, it'd be a lot more interesting comparison that Lua vs Ruby ???.

I had to reverse the dropdown values to get both the versions..

ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-linux]

Looks like perf comparison between both Python and Python 3 against Lua isn't far off from Ruby and Lua... not really surprised. Like everyone else said, Lua is smaller and leaner.

This year my new year's resolution was to learn Lua. I think with some developer attention it has the potential to become a "better PHP" for web development because it beats PHP in performance, simplicty and elegance.

Although I doubt it's going to ever replace PHP I think it's poised to make some inroads now that the next version of Apache will include mod_lua out of the box.

http://news.ycombinator.com/item?id=452396

Of course; Get a native JIT'ed version of ruby going, then you can make a fair comparison.

Unlike say...JRuby?

JRuby is not a JIT for Ruby, and it doesn't compile Ruby to Java bytecode. It's a Ruby interpreter, written in Java.

Recent work in JRuby does exactly that. It'll JIT your code, and you an even pre-compile if you want.

(AFAIK, just seen references to all this).

Cool! Looks like it's been around for over a year:

http://wiki.jruby.org/wiki/JRuby_Compiler

For what it's worth, JRuby performs a tiny bit better.

I think Ruby in general is less amenable to analysis/compilation than Lua is, so even a JITed Ruby might not do as well as Lua.

"Fair" comparisons of the sort you're talking about are not useful when choosing a language. If I'm choosing a language for an application, I don't care if it's unfair that one is much faster than the other because it has a JIT compiler instead of an interpreter - I only care which one is the most suitable.

That's like saying "Of course, get a non-sucky version of ruby going, then you can make a fair comparison."

Are both sets of code optimized?

I wonder how it would compare with Ruby 1.9 (released today)?

You don't have to wonder - all the code is downloadable. You can run the benchmark on Ruby 1.9 yourself.

That's true, but I'll live with wondering a little longer. In the meantime, it seems that Lua is still quite a bit faster than ruby 1.9.0, even though ruby 1.9.0 is approaching Python 2 territory (tested 7 months ago):

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all...

LuaJIT is x86 only, so it's not quite apples to apples.

I don't know what's going on with Ruby's 'fasta' implementation, but I'm pretty sure it's not x86 assumptions that cause Ruby to use 231x the RAM.

it's because the ruby implementation of "fasta" computes the entire result in memory and then prints it out; the lua one prints as it goes.

so, i'd take it with a huge slab of salt.

I'm sure, but luajit wins handily over all the scripting interpreters. I'm not much concerned with ruby in particular. I'm suspicious that luajit is constrained in features. I doubt you can dynamically load in object code for modules, for example. This makes it not quite a straightforward comparison to the other interpreters.

Actually, this is incorrect. LuaJIT is explicitly compatible with the standard Lua ABI for modules.

http://luajit.org/luajit_features.html

Since it is able to use certain non-ANSI C features on the various platforms it supports it can do other neat things like enable coroutine calls across Lua/C stack boundaries.

Reply on news.ycombinator.com