The evolution of Lua | Hacker News Reader

The evolution of Lua

tecgraf.puc-rio.br

92 pointsshawndumas68 comments

One serious thing Lua is still missing is native Unicode support (correct me if I am wrong).

When I discovered Lua the first time I was enthusiastic. The JIT is so fast, and everything seems to be fine. But missing Unicode is a big disadvantage for Lua. Unicode is a must to have for modern languages. It makes things so much easier when you know you don't have to convert any fonts because they are all presentable in Unicode.

After having worked with Ruby 1.8 (no unicode support in String) and Ruby 1.9 (added unicode) rather extensively, as well as C/Obj-C and a variety of other languages with varying ideas on Unicode support, I am of the opinion that Lua has the right approach. If your built-in string type does not account for encodings, you can always build one on top of it that does. If, on the other hand, your built-in type has to carry the constant burden of encoding information, you'll never get away from that (indeed, I've been talking with ruby-core about adding back a string-like library with no encoding information in Ruby 2.0).

As long as you handle usual letters it is no problem to miss Unicode.

How nice Unicode is I always realize when I copy and paste some foreign text from somewhere into Emacs or Racket Scheme and everything works just fine.

What he's trying to say is that Lua's strings are capable of storing Unicode characters (they're "8-bit clean"), but that Lua's string manipulation functions aren't Unicode-aware. If you want to paste Unicode characters into a string and print it later, Lua can handle it. If you want to do a pattern match on a Unicode string, you need to bring your own pattern matcher.

The idea of adding at least some rudimentary UTF-8 support has been gaining traction on the Lua mailing list. So maybe for the next version of Lua you'll see some sweet Unicode.

Very good!

As someone who values simplicity, minimalism, portability, and good performance, Lua is a constant source of inspiration to me. I frequently look at its design for ideas.

Please say more. What are some inspiring things you've found in Lua's design?

Some things I love about Lua:

1) Lua takes a page from the Scheme book in that the authors try to give a very minimal yet very extensible core -- they don't give you tons of features, but they try to give you the baseline constructs you need to build other features. It's easy, as it is in Scheme, to do procedural, object-oriented, and functional programming in Lua (and even mix them).

2) Lua syntax can be pretty (although I've found that it doesn't lend itself well to embedded DSLs).

3) Lua is distributed as a very small library with an intuitive (at least for me, YMMV) C API, which makes it trivial to embed.

4) The Lua VM is more performant than Python, Ruby, Perl, PHP, and Tcl (although you get far fewer libraries; no batteries included). There's also the LuaJIT implementation, which is the fastest dynamic language implementation I know of (and it's fully compatible with the stock C API).

5) Lua is straightforward to learn (at least as straightforward as Python or QBasic), yet has a lot to offer the more knowledgeable programmer.

6) The documentation is detailed, specific, and concise.

All of these points make Lua an excellent language for embedding into applications (which is what it was originally designed for), but it's not often an excellent choice for general-purpose programming or even general-purpose scripting. Point 1 is a double-edged sword, just like it is in Scheme: you often end up with many different object systems, module systems, etc. in different codebases which can be painful for interop. I hear that this is also a painpoint for Tcl.

I'm having trouble answering your question eloquently. I'll think about it more and hopefully write a blog post about it before long.

Please do. I'd like to read it.

It seems to me the outstanding feature of Lua's design history is its designers' determination to make the core ever simpler and more general. That takes vision and courage over a long period of time. What emerged is truly interesting and inspiring (much more so than cobbled languages that grow only by accretion of features) and is likely to grow in importance.

Yes, that is very much how I feel about it. They boil every concept down into its purest essence, and always avoid having two abstractions when one will do. Other languages grow when they need new features, Lua refines.

But in a blog entry I'd like to address several of Lua's core abstractions and how they are cleaner and more general than competing languages. For example, the Lua thread data structure encapsulates all necessary state for a thread, but is general enough that it actually allows for multiple thread execution models. By default only coroutines are provided (which use the thread data structure). But you can implement lua_lock()/lua_unlock() to have a Python-like model with preemptive threads and a GIL. Or you can use lua_sethook() to implement a "green thread" model where multiple pre-emptive Lua threads run inside a single OS thread.

Three options for threading models (probably more if you dig in), one core abstraction and implementation. Remarkable!

I wonder what examples one could dream up where manipulating that thread data structure in an application-specific way would be a win. i.e. some kind of domain-specific threading model.

Manipulating the thread data structure directly would violate the encapsulation of Lua and prevent you from being portable to eg. LuaJIT. But there are still lots of opportunities for building custom threading models; see:

http://kotisivu.dnainternet.net/askok/bin/lanes/comparison.h...

Then you should put your blog on your profile :)

If you enjoyed this, you will probably also like the same author's "The Novelties of Lua 5.2"[1].

[1] http://www.inf.puc-rio.br/~roberto/talks/novelties-5.2.pdf

I absolutely love how they are making such major changes as late as version 5.2. I also love that one of them is goto.

Lua is terribly impressive. Someone needs to make a transpiled language targeting Lua and JS. They're semantically close enough that this might not be too complex. If it worked well, it would be an interesting alternative to Node.js, since you could write the server-side for Lua coroutines and so on. Not to mention its performance.

Lua does have lots of metaprogramming that js lacks, and adding into js could be tricky since you'd essentially have to rewrite existing js code into new code that allows this.

However, the grammar rules are fairly convertible. I've converted a parser that normally targeted javascript, to target lua instead (it was for a project to convert lua code into json).

The easy way would be to make a language that compiles to the semantic intersection of Lua and JS; that is, don't expose things that you can't easily generate both Lua and JS for. Such a subset would cover approximately everything I for one want to do in JS anyway.

You could make it easy to take advantage of Lua-only features like metaprogramming by denoting them server-side only. You'd have to do that to get coroutine support anyway.

The idea seems so exciting and obvious that I wish I had time to do it. Lua doesn't seem to be used much for server apps, though. Why? Are there technical limitations that make it unsuitable, or is it just that it hasn't been exploited yet?

Another thing I'd like to know, given how embeddable Lua is, is whether one could embed it as a browser plugin and then get the power of Lua client-side as well. Obviously, one wouldn't demand this of users, but it would be a pretty sweet option to offer: "install this little plugin to make the program faster and more responsive".

One final thought. I wish Google had gone with Lua instead of making Dart. Lua is basically JS done right. Seems to me they could have built a compelling case around it.

I had this idea too -- I wanted to share code between command line clients, web clients and servers. Having a language that compiled to both Lua and JavaScript seemed eminently doable because their semantics are quite similar (and even the syntax is pretty similar).

The thing is that Node.js has kind of solved this now, as long as you can put up with JS. JS is ugly, but Lua is also not without its annoyances (1 indexing, and to me begin/end is a big pain over {}).

v8 probably doesn't perform as well as LuaJIT, but it's by far good enough for most purposes.

JS is ugly, but it's even uglier to have to worry about the transliteration to 2 different languages. 99% of it will work, but it won't be "write once run anywhere" -- we all know what a fallacy that is.

But since CoffeeScript became popular, I wonder how easy it would be to port a subset of CoffeeScript or a slight alteration of CoffeeScript to Lua (or even Python). I've seen MoonScript (http://moonscript.org/)

A third thing is that libraries would be totally different. You would have to bind the same libraries to JS and Lua to get any kind of portability. In the end I think that's not worth it for any project.

There's a node.js style API for Lua, but again it won't be "write once run anywhere".

https://github.com/luvit/luvit

A last motivation was the C API of Lua, which is nice and simple and portable. It seems way easier to work with that extending Node, which is tying your code to a particular JS implementation.

I knew someone else had to be thinking about this!

The thing is that Node.js has kind of solved this now

The question is whether Lua's advantages over JS (more powerful, faster, excellent FFI) can be exploited server-side to create significant value beyond Node.js. If the answer is no, the idea is pointless.

Lua is also not without its annoyances (1 indexing, and to me begin/end is a big pain over {}).

1-indexing is the sort of thing that could screw up cross-compilation. That's a worry. Begin/end - I agree with you, but don't have to deal with it because I write my JS in Lisp. Personally what I have in mind is a Lisp that compiles to JS and Lua.

it's even uglier to have to worry about the transliteration to 2 different languages.

Right. The idea lives or dies on how bad the semantic delta is between Lua and JS (in some subset of both languages chosen for compatibility). The impedance mismatch has to be really small to make it worthwhile. It's not obvious if that's the case.

But since CoffeeScript became popular

I don't see how the impedance mismatch argument is any different for CoffeeScript/Lua than it is for JS/Lua.

Projects that don't depend heavily on client-side JS libraries wouldn't suffer too much.

There's a node.js style API for Lua, but again it won't be "write once run anywhere".

I don't find it very interesting, since you can just use Node.js for that style. The goal here isn't to cross-compile to Lua and Node.js. It's to cross-compile to Lua on the server and JS in the browser.

A last motivation was the C API of Lua

Right, that's potentially a big deal. For example, you could run your server app in Lua embedded in Nginx. I know people have been working on this.

Yup. For semantics, I'm sure you can find some corner cases where closures behave differently, and the prototypal inheritance also probably differs in some significant ways. I haven't thought about it in awhile.

I'm not concerned too much about performance because I don't think Python is too slow 99% of the time, and node.js is faster than Python.

So personally I don't think there is a strong motivation anymore, but if anyone tackles it I'll try it out for sure :)

Another motivation I forgot to mention was sandboxing. Actually that was motivation for the same idea with Python <-> Lua. I have a ton of of Python code lying around. A lot of it can be extended by the user. I want to expose some of these things as network services, but it is hilariously insecure to accept Python code from users in any way. I also have a bunch of coworkers who already know Python. So it would be cool if I could somehow have a sandboxed Python interpreter (this has been talked about MANY times on pyhon-dev, etc.; not doable with CPython).

One way to get that is to try to compile Python to the Lua byte code (not portable across versions), or to Lua source code. I think the semantic delta is even higher, so it's probably not going to work very well. You would end up halfway to writing a Python interpreter in Lua (e.g. semantics of dictionaries)

PyPy has some sandboxing I hear but it's too heavyweight/early right now.

Lua is really nice. I want to use it, and has advantages over Python and JS, but I can't seem to justify it for any project. The network effects of tools/libraries in Python and JS always win out.

You can use Lua to generate content for Nginx.

I read the web pages about that a while ago. Do you know if people are using it?

If you look at the mail lists, the devs are quite keen. I think some of them have an Alibaba background, so it's probably used there.

One thing I didn't like after a quick look at Lua is that all variables are global by default.

    j = 10         -- global variable
    local i = 1    -- local variable

( from http://www.lua.org/pil/4.2.html )

Same with functions. To declare a function that is restricted to the local scope, you must explicitly use "local":

    local f = function (...)
      ...
    end

Can any Lua aficionados explain why this is the case? Surely a better design would be for variables to have local scope by default, and to require an explicit "global" prefix to be globally visible? Does this ever cause problems in real-world code?

I did a fair amount of real world Lua a few years ago. This issue did start to bother me after my codebase grew. It is fairly easy to work around, mainly due to the very powerful metaprogramming facilities in Lua. See http://lua-users.org/wiki/DetectingUndefinedVariables. I ended up adding a few lines of code to my own common library module which disallowed globals unless specifically declared with my own "declare" function.

There's a page on the Lua users wiki that neatly summarizes the various arguments: http://lua-users.org/wiki/LocalByDefault

Essentially, instead of asking "why global by default", you should have to explain "why local by default is a good idea". In practice, I've found it something easy enough to get over. You don't have to believe me, though...give the Lua Missions a try and see if you don't like Lua just a bit more than you thought you might: https://github.com/kikito/lua_missions

Local or global by default is the wrong question to ask, because implicit declarations are a bad idea either way.

Io - another minimal language - avoids the issue by having different operators for definition (:=) and assignment (=).

This discussion comes up on the mailing list occasionally.

Well, the first answer to why we have global by default is history. Lua started out as a data description where global by default was a convenience.

Local by default doesn't work so well either with nested scopes. Suppose we do have local by default and then consider this code:

  x = 0
  function a()
    x = 1
    function b()
      print(x)
    end
  end

It is impossible to refer to a's x variable. This makes it harder to return a simple counter function, for example. This is regular Lua:

  function create_counter(initial_val)
    local count = initial_val or 0
    local print_count = function() print (count) end
    local increment = function() count = count + 1 end
    return print_count, increment
  end

Every time you call make_counter() you get a new pair of functions with their own internal count variable.

See also PEP 3104 for more: http://www.python.org/dev/peps/pep-3104/

    Every time you call make_counter() ...

create_counter()

If, local was the default, you would have:

    function create_counter(initial_val)
      count = initial_val or 0
      print_count = function() print (global count) end
      increment = function() global count = global count + 1 end
      return print_count, increment
    end

Something like that, it doesn't seem that much worse. Also, why wouldn't the print_count function inherit the scope of the containing context so you wouldn't even have to have the "global" keyword.

    function create_counter(initial_val)
      count = initial_val or 0
      print_count = function() print (count) end
      increment = function() count = count + 1 end
      return print_count, increment
    end

  ... create_counter() ...

That's what I get for writing code on a small phone screen. :-)

    function create_counter(initial_val)
      count = initial_val or 0
      print_count = function() print (global count) end
      increment = function() global count = global count + 1 end
      return print_count, increment
    end

You might not want to use 'global', because that implies global scope. And in this case, this is not what was intended. For Python, keywords like 'outer' or 'nonlocal' were proposed.

All in all, I'd be OK with 'nothing by default', but I'm also OK with global by default as it is now. If accidental creation of globals is a problem I can use the 'strict' module.

    function create_counter(initial_val)
      count = initial_val or 0
      print_count = function() print (count) end
      increment = function() count = count + 1 end
      return print_count, increment
    end

If local was the default, wouldn't the assignment to count in increment() create a new local variable (which is then discarded)?

The reason is legacy. I personally lint all my code to prohibit global writes, and I wish I wouldn't have to.

Hopefully by now language designers realise that anything-by-default is wrong. (That's right, I'm looking at you, Python.)

What linter do you use, and does it have vim integration?

I wrote my own. It basically imposes some restrictions on the way you are allowed to use Lua with respect to globals (with escape hatches). It also hooked into our Python-like import system. It checked illegal global writes and reads, illegal imports, unused imports and unused locals.

I also thought that this was a problem until I used Python and the "nonlocal" keyword didn't exist.

I was surprised to learn that coroutines were new in 5.0 until I also learned that 5.0 was released in 2003.

Interesting for me to know that Lua has been used seriously for game development. What are the main limitations?, how come people aren't developing more in Corona rather than straight objectiveC. I'm guessing the Corona environment itself is the limitation?.

You just said that you've just learned about lua used in games, and then directly propose Corona as the only place where you can use lua.

You can embed yourself lua, and that's it's power. It can go down to 30-40kb (without parser), and run on PS3 SPU chip (256kb memory).

It does not use any globals, all goes through a state structure, so you can have multiple runtimes (and different ones, HavokVM, lua reference, luajit) if you want at the same time.

Don't get me wrong, Corona is cool, but there is also moai, and tons of other toolkits, and hundreths of other games that are using it - for scripting, ui layout, configuration, many other things.

Lua is pretty minimalistic. There are few batteries included. The standard library is very basic. There is no native object system, though implementing one is trivial.

That said, I friggin love Lua for its flexibility and transparency. Once you grok its few basic mechanisms, everything else just falls in place. In contrast, understanding Ruby or Python is a lot more difficult since the languages themselves are a lot more complex.

In fact, if I were to teach programming, I would teach it using Lua. Implementing your own object system is increadibly instructive. Learning about coroutines and lambdas is, too. And Lua is still small enough to just read its source when in doubt. It's bytecode consists of just a few dozen instructions.

Honestly, Lua is a marvellous language and I just love to use it!

Do you have any recommended resources for programmers looking to pick up Lua?

Programming in Lua seems to be the best choice, but mostly by virtue of the very small number of coices. It was most recently printed in 2006, but there were few breaking changes since then, and you won't run into them very often.

To be honest, I learned it using only the tutorial here: http://lua-users.org/wiki/LuaTutorial and google.

Roberto Ierusalimschy's Programming in Lua, 2nd Ed. is an absolute gem. Well written, concise and very authoritative (he's one of the language's co-creators).

One thing worth knowing though: he's working on the 3rd edition (updated for Lua 5.2), though I'm not sure how soon that is expected.

The first edition is online, if you want to get an idea what it's like: http://www.lua.org/pil. In addition, the language's refrence manual is freely available online as well. Depending on how you learn, that would get you started very quickly. (It's a small language and very much follows the principle of least surprise - except for indexing arrays from 1, but you get used to that.) http://www.lua.org/docs.html has the full reference manual for 5.1 and 5.2.

Seconded, the book is friggin awesome, if a bit dry. It is written for an experienced programmer though.

Personally, I learned object orientation through this book. I knew basic OO before, but this book really explained it to me.

>In fact, if I were to teach programming, I would teach it using Lua. Implementing your own object system is increadibly instructive. Learning about coroutines and lambdas is, too. And Lua is still small enough to just read its source when in doubt. It's bytecode consists of just a few dozen instructions.

If these things appeal to you check out lisp (clojure is my personal recommendation)

Lua itself can stand pretty well, especially with LuaJIT2. That said, I'm starting to learn Lisp right now (Clozure/OpenMCL), and coming from primarily Lua development in the last two years, it seems much less like the leap many play it out to be..

Instead, it has seemed more like simply stepping from a clean, compact room into a large, similarly clean and smooth room but with depth you hadn't imagined before.

Well, that and learning Emacs compared to ST2 is a bit of a step at the same time.

I know a bit of eLisp and I quite like it. Frankly, it's not that Lisp syntax is bad (it's wonderful), but most popular languages have a C-derived syntax. For teaching to non CS-types, the target is probably a C-like language and I guess Lua is closer to that than Lisp.

If I were using a Lisp these days, I'd really miss the Lua table.

Lisp is built around cons cells and the linked list. But it has seperate containers of various types.

Lua is built around the Lua table, which can function as a list, array, and dictionary/hash table. You can then build any other data structure from tables easily. With a metatable, you then also get read-only tables, objects and classes, and more.

Now if someone were to design a modern Lisp based around tables...

>Now if someone were to design a modern Lisp based around tables...

Check out clojure - advanced data structures/primitives + immutability(functional programming)

I don't really want multiple data structure primatives though, just the one. And have the language built up around it.

I have been mulling over better support for immutable data structures in Lua, but I haven't figured out a good and clean design yet.

Now if someone were to design a modern Lisp based around tables...

I agree, it's an intriguing possibility. There was a thread about this a while ago:

http://news.ycombinator.com/item?id=3291065

Lua is pretty minimalistic. There are few batteries included. The standard library is very basic.

To what extent does its easy C interop make up for this?

It helps quite a bit. It's really easy to write a glue layer between Lua and whatever C library you want. The downside is that you still have to actually write that glue.

It should probably be noted that Lua gets along less well with C++, so that glue is a bit harder to write for C++ libs. There are tools that will automatically generate that glue for C++, but I'm not familiar with them.

As Dyson said, it's fairly easy.

LuaJIT2's ffi also makes it a lot quicker.

I tried to get into Lua, but with issues with installing LuaRocks and the library I was using, LuaFileSystem, I wasn't all that impressed. This could most likely be the library I was using or I didn't give it enough time. So, could you go into more detail about Lua's flexibility and transparency and what is it that you grokked about the few basic mechanisms? What about library support? It seemed that LuaRocks was the most popular module manager and it didn't seem to have that many libraries, and some seemed to be orphaned. My question is if you have experience with this or is this a mischaracterization of things?

I wanted to start out using Lua as a scripting language which may not be a good fit, but it seemed more compelling than other languages I was looking into.

Lua is indeed widely used in games. The plus is that it's a very simple language to pickup and anyone on the team can learn to use it. The downside is that anyone on the team can use it. :) Memory is an extremely rare commodity in embedded systems and Lua loves to gobble it up.

Consistent perf in video games is crucial and the Lua garbage collector can be brutal in this regard. Newer versions of Lua support incremental garbage collection but it's quite realistic to dedicate 2-3 milliseconds of every frame to garbage collection. Yuck. (16ms = 60fps, 33ms = 30fps)

Lua is trivial to implement into an existing engine. Lots of iOS engines heavily rely on Lua. Large games devs don't want to become reliant on those middleware engines. It can take them many months to update to latest iOS sdk. It took several popular engines 6+ months to integrate the in-app purchase SDK update.

Havok, owned by Intel, purchased Kore and turned it into Havok Script. Havok Script is a virtual machine that runs a "Lua compatible" language. It is extremely fast. Ridiculously faster than straight Lua. It also has full debugging support and you can trivially step through a call stack that bounces between C++ and script. Havok middleware is not known for being cheap.

luajit2 in interpreter mode is also very fast, originally written in x86 (Mike Pall has probably finished the arm port by now). But I've never heard of any benchmarks between the luajit2 interpreter v. havokVM, that would be interesting to see.

Keep your eyes peeled for a port of the LuaJIT interpreter to the consoles. From recollection it's two or three times faster than straight Lua.

I think you'll find many if you look. It occupies a nice slot between the speed and control of C and the ease (and slowness) of python, java etc. Codea, app development in iPad for iPad uses lua. It was recently mentioned on HN.

Computercraft, a mod for minecraft embeds lua in minecraft for you to program. Not a direct use of lua in gaming, but cute!

Lua is used largely in AAA development. Most famously, it powers the UI for World of Warcraft, and users are able to write their own complete UIs in Lua.

Many other games use it for scripting. When I worked at Sony, we were using it on quite a few PlayStation games.

I have such a strange love for Lua. It was one of the first languages that I learned and used to program homebrew for the PSP. It's really where my journey as a programmer began and was the first language that I programmed with for fun. I don't think you can ever stop loving the first language you got your hands dirty with.

Unless of course it's C++

</personalopinionwithouttryingtoinstigatereligouswar>

Technically, my first programming language was C++, and I really don't love it. ;) But that might not count: I actually started with a markup language, HTML, which I use daily.

</sameDisclaimer>

If you want to tinker with Lua, Codea on the iPad lets you code in it and create little iPad apps (for your own enjoyment).

Warning: this links to a PDF.

Usually PDFs get reposted to Scribd or a similar service - it's somewhat unusual that it wasn't in this case.

Strange but much appreciated!

Perhaps the trailing octothorpe (#) threw off the HN functionality that does this. Noteworthy.

Reply on news.ycombinator.com