Compiling to Lua | Hacker News Reader

Compiling to Lua

blog.cloudflare.com

115 pointsjgrahamc35 comments

I can't help but feel like Lua is one of those languages that is fantastic, but gets basically ignored for most things outside of a particular niche (game scripting). Perhaps that is why it is still such a fantastic language...

I hope that Luvit[0] is what will bring Lua to the forefront of web development. It has such remarkable potential. It could completely (and should completely) replace node.

[0] http://luvit.io

It seems like a step backward. Dealing with callbacks manually is an unnecessary hassle compared to being able to use "green threads". See python gevent and ngx_lua for examples of people doing things this way.

Check out Luv - an attempt to use Lua's coroutines to do green threads/fibers, integrated with libuv.

https://github.com/richardhundt/luv

luv does a fantastic job of sticking with Lua coroutines. The master branch still has zeromq in it, but the latest work is done in the ray branch. Sadly, it doesn't seem the project is moving, so most of the libuv features are not yet implemented the "ray" way.

Edit: Seems ray does have most of libuv features, sorry.

and node-fibers :o

Yeah I never really understood luvit. Callbacks make sense in JavaScript because it doesn't have coroutines. But Lua has coroutines! So why does luvit use callbacks? It does seem like a step backward.

Historical reasons. The guy who wrote luvit knew NodeJS but didnt know Lua when he started working on it.

It's the other way around, it gets ignored because it is consciously designed and maintained to serve a very particular niche: high performance extension language. I wouldn't consider Lua a fantastic language anywhere else.

I`ve had the opportunity to use lua to program the state machine of a protocol and I must admit I was really blown away by the performance.

It's designed as a simple and performant language. Embedding within games, for example, is one area where those traits are very valuable. That doesn't mean Lua is only good for embedded programs or games: Lapis and Luvit are testaments to how Lua is great for building web applications.

On top of that, LuaJIT means it blows conventional languages like Ruby and Python straight out of the water. Not only is it a good language for web development, it's a remarkable one.

Could Python or Ruby be compiled to Lua? Such languages are hard to compile to static typed languages like C, but Lua is dynamic like they are, which should remove a lot of problems.

Sure they could, someone just needs to write the compiler for it. There may already be a project that does this anyway. IMHO, the most challenging part of this would be that Lua doesn't have as many libs as Ruby and Python. So the compiler would have to do a lot more work rather than be just a syntax converter.

Technically yes, but it would likely only be possible with a restricted subset of the language. Python and Ruby have a large standard library and set of built-in functions, because they're "batteries included" languages. Lua is quite the opposite: it doesn't even have access to the file system unless you implement that or use a module. So you'd either end up using a limited Python without features absent in Lua, or you'd need to bridge the gap in functionality with a very large runtime library.

If you're looking for a more expressive syntax, check out MoonScript. It's an awesome language that you could consider Lua's CoffeeScript analogue.

Aren't a large part of the Python and Ruby standard libraries written IN Python and Ruby, respectively? That should greatly reduce the amount of work.

Lua does have file system access: http://www.lua.org/manual/5.2/manual.html#6.8

In that it can open files. It does not, however, have a way to enumerate files in a given directory---that requires a module to handle.

Ah, true. Penlight helps, but you are correct it is a separate module. I think for being the target of a higher-level language it might help that you aren't married to a standard library.

I fully agree with you but just want to add that you can do it in "pure" Lua :

for file in io.popen("ls "..dir):lines() do ... end

This should never be used in real applications and a proper filesystem module should be used instead. But, I find it very useful for quick one-shot scripts I write for batch processing some files.

Lua has access to the filesystem, but you probably want the luafilesystem module for general scripting, it provides more POSIX and Windows-specific functionality.

As a general rule, the Lua standard library only includes what can be portably built on the C standard library. Instead of "batteries included", Lua is small and portable everywhere ("travel light"), and makes it easy to wrap other functionality via the C API. Sometimes that's a problem, but sometimes it's exactly what you want - say, when Python would be too big a dependency, Lua is an excellent option.

(No experience with MoonScript. I'm content with Lua's syntax. I much prefer it to Ruby's, for example.)

Yes. I once thought about cheating my way to "the fastest Ruby implementation ever" by simply compiling Ruby to Lua and handing the result to LuaJIT.

Many people "compile to C" if they just want to write some personal toy language or an experimental language. I think "compile to Lua" would be a superior alternative in many cases.

One should point out that this way you get an incremental garbage collector for free, one whose behavior can easily be controlled at runtime i.e. good for soft real-time applications (e.g. games).

Also both Lua and LuaJIT are very lightweight which means you can simply bundle the complete language implementation when creating standalone binaries. On Windows x86 Lua5.1 is ~308kb compiled, LuaJIT2 ~340kb.

Have you heard of Terra[1]? I think it was pretty much designed for what you're saying. :-)

The cool thing is that the project looks really active too.[2]

[1] - http://terralang.org

[2] - https://github.com/zdevito/terra/commits/master

One could make a Lua backend to PyPy? :) Lua 5.2 has a nice `goto` which makes things very easy. Esp with builtin TCO.

I have experimented transpiling PHP to Lua and it worked quite well. LPeg makes things very easy for parsing and the all-tables style makes it fairly easy to model any kind of class/OO.

Not really. If you want to compile down to Lua you have to either :

1. Accept Lua semantics (for example, numeric semantics) 2. Accept a (severe) performance hit in thoses cases where you want to have different semantics.

Disclaimer : I actually had this very idea and tried to make a Python to Lua compiler. Since my reason was to see if Python on LuaJit could be faster that CPython, i quickly realized this wasn't possible if you wanted to fully respect python semantics.

OTOH you could do a language that is very close to Python semantics and very close to Lua(JIT) performance, if you accept to deviate in a few points.

Numbers: there's no issue with handling numbers using userdata. See how lgmp handles this [1]. There's a wider issue of just how Lua is the target language if most of the runtime and much of the semantics is being done in C libraries...

Which Python constructs in particular did you think were tricky? I had thought a bit about the special case of modelling Python comprehensions and decided that they translated nicely, but I might be deluding myself since I implemented nothing.

[1]: http://members.chello.nl/~w.couwenberg/lgmp.htm

Implementing a Python-ish language is easy enough. Depending on your definition, Lua already is one. Implementing the totality of Python is very hard to do performantly. Anything you think you can cache can probably have that cache violated, and real code will tend to do it. For instance, surely you can just cache a class -> method table once? My goodness, no. The class can be dynamically modified, including rewriting what methods exist. This happens in, for instance, interpreter reloads, so it's a real problem. An instance can have a method dynamically overloaded. (Prototype inheritance is not supported by the language's style or keywords, but it is essentially supported by the semantic model.) Heck, an instance probably isn't what you think it is, check this out (tested in Python 2.7.4 and 3.3.1):

    >>> class A:
    ...     def method(self):
    ...             print("A")
    ...
    >>> class B:
    ...     def method(self):
    ...             print ("B")
    ...
    >>> a = A()
    >>> a.__class__
    <class '__main__.A'>
    >>> a.__class__ = B
    >>> a.method()
    B

"Aaaaaaaaa!" goes the interpreter implementor.

It's not the raw syntax that is hard, it's the mutability of everything, you know, a user gets an instance of something, adds an overload of the __add__ operator to just that instance which then proceeds to change the class of the other element of the addition operation... sure, it doesn't happen often but it happens often enough you can't just ignore it, unfortunately, or what you've implemented isn't Python anymore.

I'm convinced that this is a hard task, hard enough that probably no one will do it properly. But for emulating Python class semantics, is there any reason why we can't:

1. Come up with an inefficient but exact modelling of the semantics in pure Lua using tables and metamethods. I think I can write metamethods showing exactly the behaviour you describe.

2.Come up with an efficient implementation of python_class userdata?

The problem you describe is the problem of really understanding what's going on, not a deficit in Lua's ability to model it.

Its very hard. This post is about compiling a much simpler language with well defined semantics. Ruby and Python are simply not very well defined or simple, unlike Lua, which is why PyPy is a much more complex solution than LuaJIT (indeed Lua was chosen for LuaJIT for these reasons). Simplicity has performance benefits.

I'm using it for scripting/configuration of my mail-client. Which is pretty cool.

is cloudflare using openresty( its author agentzh works for cloudflare)? or their own nginx + lua implementation?

We use openresty and, yes, agentzh, the project's creator, is on our team.

why not pay him to get websocket support out? who prioritizes his work queue?

What a bizarre comment. They are paying him.

He has said that someone could sponsor him to do it in his own time https://github.com/chaoslawful/lua-nginx-module/issues/165

He prioritizes his own work and I trust his judgment.

Reply on news.ycombinator.com