State of the Lua Ecosystem (2013) | Hacker News Reader

State of the Lua Ecosystem (2013)

speakerdeck.com

87 pointsfuzzythinker58 comments

To get conversations started..

Lapis[1] on Openresty[2] is one of the most simplest, documented, and performant full web framework stack out there, yet almost no one knows it.

Hello World Open[3] - 2.5k teams, only 17 ( < 0.7% ) teams used Lua. This is an open competition, which means people can use their favorite language instead of Java, and less than 1% uses Lua.

The lua community really need to start doing something about this.

1. http://leafo.net/lapis/ 2. http://openresty.org/ 3. https://helloworldopen.com/teams

> The lua community really need to start doing something about this.

Why? A small community has advantages, too.

Could you name three?

- Can make backward-incompatible changes (see: Javascript, Python)

- Less trolls & nasty people in general.

- All the advantages of a niche language (I got my current job because no one else in my country knows Lua :P).

Advantage 1 is the main reason why today's Lua is not as complex as today's JavaScript, despite Lua being older.

It also significantly helped that Lua's main users could easily use an older implementation if something they depended on/didn't feel like fixing was removed. In contrast, breaking backwards compatibility for Javascript would be unthinkable.

OTOH, there is real fragmentation between 5.1 and 5.2, so not even Lua is immune to the downsides of breaking backwards compatibility.

I wrote Lua Toolbox in Lapis mostly to experiment with it. I think the framework has a lot of potential but I have two main issues with it:

1) It is written in Moonscript, which makes understanding its internals harder for Lua programmers. On the application side, Lua support used to be a second thought but now I think you can do almost everything without writing a single line of Moonscript.

2) It is tied to OpenResty. This is both a good and a bad thing, because it means you cannot use several of the existing libraries in the Lua ecosystem.

If you are looking for alternatives, a new framework called Sailor was presented at Lua Workshop 2014 (slides: http://www.lua.org/wshop14/Dalcol.pdf)

Lua has 15 teams (not 17, since it's not a lang query, but a full text search) by looking up `document.getElementsByTagName('tbody')[0].children.length` after looking up a language (on this page: https://helloworldopen.com/teams )... for all the not-so-popular languages:

    Lua: 15
    Haskell: 75
    F#: 13
    Rust: 4
    Clojure: 94
    Scala: 82
    Erlang: 6

Also, I thought these 2 would've got more teams:

    Perl: 2
    coffeescript: 28

I'd say that Lua is not doing so bad after all

To me, the fundamental problem is that Lua is designed to be an embedded extension language. But the vast majority of the publicly available extension modules are provided as shared libraries. If you have embedded Lua in your application, you are forced to use dynamic linking to use those modules. And if you use an OS that doesn't gracefully support dynamic linking (like uClinux on some platforms), you are screwed.

A second problem is that those modules tend to be kitchen sinks. For instance if you want UDP networking, and pick one of half-dozen modules that provide that, nearly all of them also provide other things you don't need. Why do people choose Lua? Small footprint, often. Why then provide bloated modules?

In other words, the module ecosystem only addresses one use case of Lua - as an extensible scripting language - and misses all of the others. And the end result is that it actually has Zero utility for most Lua users.

A lot of modules are pure Lua. They may depend on a few critical C libraries though. But with just LuaSocket, LuaFileSystem and maybe LPEG you can use most modules.

There are several ways to build those critical Lua modules statically, one being https://github.com/stevedonovan/luabuild Even without those, it is not very hard to modify the makefiles to build statically.

About the bloat: this is certainly true for some modules and not others. As I said in the talk, everybody uses Lua differently and nobody will use all the modules and tools available, but there is still some amount of code that can be shared.

I mean: you could say exactly the same thing about C, and it still has a lot of libraries.

the Lua interpreter is so small and simple that if you really needed libraries and couldnt' dynamically link them in you could easily compile them into the lua interpreter and push the module onto the global table :/. It's really not hard.

> To me, the fundamental problem is that Lua is designed to be an embedded extension language.

This. When people like myself want to use it outside that small goal we end up switching to python or go despite how amazing projects like openresty are.

It is not a small goal; it's a different goal. You should use the right tools for the right job. If you need a general purpose language, use one.

That's the contradiction I wanted to underline with my two first sentences. When you browse the community-run Lua Wiki, Lua looks like a general purpose scripting language. But not everyone agrees on this. I see no reason it cannot be both.

As someone said in this thread, with so many uses cases how can we hope to build a useful ecosystem? I think the first step is to acknowledge this community fragmentation and deal with it.

> When you browse the community-run Lua Wiki, Lua looks like a general purpose scripting language.

I don't agree with that.

> I see no reason it cannot be both.

There are several. The first one is that some design decisions in the language (like restricting all the C code to C99) rule out some features which would be considered basic in a general purpose language (like listing the contents of a folder, or using sockets). These decisions make it good in other areas (like running in a very memory-constrained environment, like an embedded microprocessor). Making it fully-featured risks throwing all that away.

The second one is that its creators don't want to transform it into a general purpose language.

> I think the first step is to acknowledge this community fragmentation and deal with it.

I am sorry, but what you are referring to is not community fragmentation; it's people who are familiar with the language, on one hand, and people who ... are not. And thus come into it with the wrong expectations. I'm not saying this to be offensive.

If you want a language which is at the same time embedded and general-purpose I think your best option is looking elsewhere. I've heard good things about mruby, perhaps it'll interest you:

https://github.com/mruby/mruby

s/99/89/ :)

I would not be so radical about Lua not being fit for general purpose use. Historically turning Lua into a general purpose language has been the goal of the whole Kepler project. Whether they succeeded or not is a valid question...

And now with LuaJIT there is a clear incentive to use Lua as a general purpose language: it is the fastest dynamic language available. At my previous company we used a lot of LuaJIT on the backend, and I would say it is very doable with the help of a few popular libraries (Penlight + LuaSocket is a good base). The FFI helps a lot too (and makes it an incredible glue language: you don't even have to write bindings anymore).

I still think Lua is not the best language to write Web applications. We used Python and the Pocoo stack (Flask...) and there is nothing so advanced in Lua (yet?), although frameworks like Lapis have to be considered. But for other things Lua(JIT) is a very decent standalone language.

Note: I don't know if everyone realizes this or not but OpenResty is an embedded use case. You are running within nginx.

> To me, the fundamental problem is that Lua is designed to be an embedded extension language.

I see that as an advantage. We already have plenty of general-purpose languages.

> If you have embedded Lua in your application, you are forced to use dynamic linking to use those modules. And if you use an OS that doesn't gracefully support dynamic linking (like uClinux on some platforms), you are screwed.

Even if you are in that particular tiny niche, you are not screwed. You can still download the source code and recompile (which will be second nature for you, if you are doing uClinux).

> A second problem is that those modules tend to be kitchen sinks. [...] Why do people choose Lua? Small footprint, often. Why then provide bloated modules?

Some modules are like you say. Others aren't. I am very careful not to put more than is necessary in my modules.

> In other words, the module ecosystem only addresses one use case of Lua - as an extensible scripting language - and misses all of the others. And the end result is that it actually has Zero utility for most Lua users.

Even if that was true, you are assuming that using it as a scripting language is not the main usage. Maybe it is, and the "other cases" are a small minority.

But it is not true. As others are saying, plenty of modules in luarocks are pure-Lua, and completely cross-platform.

Video:

http://2013.capitoledulibre.org/conferences/lua-workshop/sta...

Thanks for sharing. I love Lua. First worked with it about 10 years ago. Still saddens me that it hasn't seen more widespread adoption.

I think that most here are entirely missing the point of Lua. "The Lua ecosystem" isn't a thing that makes sense. Lua isn't a standalone language - it's an extension language, and is specifically popular because it doesn't have much in the way of a standard library or built-in assumptions about its environment.

If you can't make any assumptions about the environment, you can't have an ecosystem, because everybody's doing something slightly different. Some might be running every Lua script in a separate environment, some might have every Lua script bundled into the same environment, some might be running Lua on a separate thread so blocking is fine, some might integrate Lua into their application's event loop library. How are you supposed to build a library that serves even some reasonable amount of the userbase's use cases?

It goes beyond that, though. One of Lua's goals is to be implemented in 100% portable C, to the extent that it has no platform-specific #ifdef's (no #ifdef's at all, actually). This means that to use a lot of incredibly basic functionality supported on all relevant operating systems (like, as mentioned in another comment, changing directories), you need (for vanilla lua and luajit on platforms that don't support the ffi) a C library dependency, or a wrapper script (for luajit on the other platforms). This is pretty silly; any user of a platform without a filesystem would surely know that chdir would not be supported.

Beyond that kind of indefensible omission, there is much ecosystem fragmentation and duplicated effort that could be solved by standardizing tools like a package manager without requiring you to install it on embedded targets.

This "indefensible omission" is solved by using LuaFileSystem. Not having it in the core language is not such a problem. Another example is sleep(), which is not portable. You can find a sleep function in libraries like LuaSocket and luaposix.

Fragmentation, however, is a problem. The Lua authors have often refused to "bless" (term used by the community when discussing this) libraries. Roberto Ierusalimschy even said during a Q&A he'd rather "bless" a group of people to do it instead of him.

The problem is, as several people said in this thread, that it is very hard to write a library that fits every use case. Even something almost "standard" like LuaSocket is not used by everybody.

Some people in the community (including me) are trying to reduce the fragmentation. The authors of the two main ways to manage "packages" (LuaRocks and LuaDist) are working together, etc. But I think we will never manage to standardize on One True Library for everything.

I didn't say you couldn't write a trivial library to access sleep or chdir, just that I think it's ridiculous to need to do so. You may disagree, and clearly we can still get work done in the language, but IMHO Lua is suboptimally small, and the local maxima for "tiny yet useful" languages would have a lot more in the standard library and a few "blessed" tools to boot. A language with the goal of implementing in the standard library everything trivial, fungible (one chdir wrapper is as good as any other), useful for most users, and portable across major desktop and mobile OSes (embedded users will know if they don't have e.g. a filesystem) would be a lot more useful yet probably not meaningfully larger than the existing implementations of Lua. Sure, this would mean more stuff to lock down if you're using it as a sandboxed scripting language, but a compile-time flag with reasonable defaults for a sandboxed environment could be provided.

The os.execute function does all this. os.execute("timeout "..seconds) for example. More complicated for file system operations, but it's possible.

This is significantly less than optimal though.

And not portable (or safe, for things like path manipulation). But sure, you can still use os in small scripts.

Why would you want somebody to be able to chdir or otherwise manipulate global state in, say, a dialog script for a game engine, though? You probably don't. You probably don't want them to be able to do lots of things. Hence the standard library lacking most things.

Yes, having a package manager might solve some issues, but not for any of Lua's primary use cases. It's not supposed to be the language you write your web app in, or a shell script replacement, or whatever else, and the language developers really don't care about those use cases. It's not supposed to be anything out of the box aside from some syntax and semantics that are Turing-complete and reasonably usable.

Where I've seen Lua used, often even the existing standard library is modified or not used at all due to constraints from the embedding program. What point is a package manager if you can't use most of the packages in many environments? This isn't just embedded ones - this is on the desktop as well.

If you're looking for something more capable out-of-the-box, I'd go for embedding Python or Ruby. Lua simply serves a different niche.

Lack of userland packages isn't necessarily a bad thing. A well-documented, complete standard library (like in Go, for example) takes precedence over community IMHO.

Even as a Node.js dev, I don't understand the obsession with creating userland modules, and the value in modularity. Yes, a large library of modules correlate with community involvement, but it does not necessarily mean quality.

Lua's designers purposefully try to keep the language's standard library vanishingly small to reduce portability issues and the footprint for embedded targets. So for Lua, if no one makes user libraries, you don't have any libraries.

Lua's standard library is so small, in fact, that until 2012 there was popen() but no pclose(), a bizarre situation that persists today for people still using Lua 5.1 (which is probably the most popular version deployed).

So you can spawn a shell to run an external program, but you cannot retrieve its return code to know if it succeeded or failed. Now that's minimalism!

The minimalistic core language is OK if the authors don't want to maintain more. What I miss is some incarnation of the language that would still be more convenient than Perl or Python (by being a single executable) but have enough "basic libraries" inside. The last time I investigated I failed to find something like that.

Once you have Lua with the tons of the packages in different files with the package managers and stuff you lose the benefit of having a small-size-of-the-executable language. I understand that the language author doesn't want to be a part of that.

Have you tried http://luapower.com/

Thanks! If it would allow that the modules locally exist inside of a single ZIP file instead of the single directory, it would be almost exactly what I want.

Then the "getting started" would be

"download luajit

choose/download wanted packages along with their listed dependencies

unzip all over the same directory

(optional) rebuild binaries

(optional) zip all modules to modules.zip. <== new

run a demo: luajit ..._demo.lua

This will give you an instantly portable luajit distro with only a few small files!"

Java also has JAR files which are actually ZIPs.

I don't like luapower's approach to package management. Basically it stuffs the contents of packages in the base of a single directory:

  luapower/lib.lua
  luapower/additional-files-for-lib/stuff.lua

Instead of completely encapsulating each package in its own directory:

  luapower/lib/lib.lua
  luapower/lib/stuff.lua

So instead of being able to just cd to luapower and git clone repo, you need to use a silly wrapper script that overlays the repos with each other. All of this presumably because the author wanted to type `require "lib.lua"' instead of `require "lib/lib.lua"'

I like it, I'd like even one step more, of making a ZIP of all of the packages. I never modified in place any Perl or Python module, and I don't expect I'd do it with Lua, so having a small portable file set of just a few self sufficient files I could copy to the another machine is my real wish.

But it's arguably easier to package the other way I described, just stick all of the relevant directories into a zip file, instead of needing to manually pick out the "lib.lua" files from the base directory. Literally the only benefit of this approach is saving a few characters on require statements.

For anyone interested in creating a new language, make your equivalent of `require "lib_directory"' do the equivalent of `require "lib_directory/init.lua"' or `require "libname_directory/libname.lua"' so we don't have this issue.

Do you actually do that manually? Why isn't there a script for that? It should be a part of the automated process. Much better doing it once as an automatic step than writing everywhere in your sources:

somethingbig/somethingbig

every time you use somethingbig. And that step should also zip the modules, I believe.

The behavior of require, as I'd like it:

require X first looks in the directory X, for module X. If there is no such, it opens modules.zip and reads module X from there. If internally in the ZIP there were a subdir for every module, I don't care, as long as I don't have to type that fact every time.

modules.zip would be constructed automatically. Whoever wants to tweak some specific module can unpack it in the separate directory.

Having reread the Lua manual, `require "lib"' can easily be made to execute `require "luapower/lib/lib.lua"' by adding "luapower/?/?.lua" to LUA_PATH. This is the way the standard /usr/local/lua libraries are require'd. Now I cannot think of a single explanation for why luapower works the way it does.

I'd prefer modifying require to not need a new entry to the ENV var for every used module. It is a huge waste of ENV.

http://www.lua.org/pil/8.1.html

> To determine its path, require first checks the global variable LUA_PATH. If the value of LUA_PATH is a string, that string is the path. Otherwise, require checks the environment variable LUA_PATH.

Even if you needed to modify an environment variable, I still think that would be preferable to creating yet another package manager and fracturing the Lua userbase further.

As a user, I want to "just use it" and the binary to "know" where the stuff of the module is without me having to hold its hand. Specifically, I don't want to know about LUA_PATH unless to use it to point to the single dir where I put my own modules.

Your concern doesn't make any sense. Any package manager has to modify LUA_PATH in order for require to work with it in that way (unless it installs libs in /usr/local/lua, which requires admin privileges and isn't portable to Windows). luapower surely does this already, adding "luapower/?/?.lua" to the path instead of "luapower/?.lua" is a one-line difference. It's completely tangential to the criticism I gave.

So, LuaRocks contains just 330 packages and doesn't look like the best package manager for Lua (and they also have a terrible website as for me).

If a community here wants to make things better - let's meet somehow and discuss it.

Then if, say, each of 5 Lua developers converts 1 package per day in their free time then in about three months the whole list of packages will be converted.

Anybody?

Speaker here, funny this got picked up by HN over a year after my talk.

Why makes you think LuaRocks is not the best package manager for Lua? As far as I can tell, it is.

Things have changed since last year, too! My suggestion of slide 40 ("programmatic releases for LuaRocks") has been implemented by merging with MoonRocks, which is now the official source for packages.

This has accelerated the rate of packaging: as of today there are over 500 packages available.

That being said, if you want to make things better there sure are lot of things that could be done. If you want to contribute the best place to reach the community is probably the lua-l mailing list and the more specific luarocks-developers mailing list.

Oh, thanks for a prompt response! I personally got stuck with LuaRocks and cross-compilation for our MIPS board. Probably it was lack of documentation or lack of my knowledge. So instead I manually crafted most of the packages we needed either from scratch or by writing custom makefiles for existing ones (that was more than a year ago).

The whole idea of rockspec looks very nice to me and I totally support it.

But can I keep a global aka system rocktree and a per-project local trees? (Pardon my ignorance, I really don't know that). Per-project trees make deployment a lot easier.

Also, why can't "luarocks" or "moonrocks" be a single executable/script? Isn't it easier to get user just a download link instead of: "download it here, unpack, configure, make build, make install, now use it to install moonrocks"? (Also, I'm not saying that it's a very complex procedure, I know most people can do it easily, but that leaves an impression that the LuaRocks project is not very user-friendly).

Note: I am not the author of LuaRocks (https://news.ycombinator.com/user?id=hisham_hm is), just a user.

Regarding distribution, what OS are you talking about? LuaRocks is available in Homebrew for OS X and as a package in most Linux distributions. Windows build looks a bit more painful but I don't have experience with that OS.

Regarding per project trees, there is no such thing as what exists in Python with virtualenv, but you can certainly do this:

    $ mkdir my_project
    $ cd my_project
    $ mkdir rocks
    $ luarocks-5.2 --tree="./rocks" install haricot

This will install the latest version of Haricot and all its dependencies (in this case LuaSocket) in the local tree regardless of whether you have them system-wide. Note that you can change this behavior (http://luarocks.org/en/Dependencies) but the default dependency mode is "one".

LuaRocks dev here, thank you for the feedback!

> But can I keep a global aka system rocktree and a per-project local trees? (Pardon my ignorance, I really don't know that). Per-project trees make deployment a lot easier.

Yes, catwell's comment expanded on it below.

> Also, why can't "luarocks" or "moonrocks" be a single executable/script?

The documentation on the website is outdated. Nowadays the "luarocks" command has absorbed the functionality of the "moonrocks" script. Just use "luarocks upload foo-1.0-1.rockspec".

Even worse!

Lets look at the Orbit web framework. This framework was (and likely still is) broken because a dependency to a pattern matching library that changed. The patch is only a few lines, but a door stopper for anybody who tried Lua for the web.

Now lets look at a simpler problem. Your script wants to change directory, but Lua is lacking a chdir function. Google will tell you, that this is in some POSIX library, and Debian or your distribution of choice of course offers a POSIX package for its Lua. Soon you start to wonder: Is there any documentation, what else is implemented in this posix lib? You find no docu, instead you find dozens of Lua posix packages, all without documentation and all without regression tests all over the web. Wondering, what did the maintainer of your distribution of choice pick?

Praise the Lord! The church saw that module system was bad, and decided to remove the module keyword from the language with Lua 5.2 to get rid of all those bad modules :D

imho, there is a cultural difference. There is more then one way to do it in Perl, exactly one way to do it in Python, but in Lua you have to DIY.

Lua has two strongholds, where it excels: Education, where its nice, that the teacher has an excuse to dig deep into C bindings, just to implement a chdir function. And in systems where Lua acts as a glue language. A module system would be counterproductive in both cases.

zserge> Anybody?

I'm hanging around in #lua on irc.freenode.net for a chat.

Luarocks is a clunky package manager. But its biggest crime is that packages are neither signed nor are they transported over https.

http://luarocks.org/en/Documentation

That said, some of moonrocks goes over http, https://rocks.moonscript.org/

The only excusable way for packages to go over http is if they were signed and the signing chain is secure.

Do you think this is a factor that slows adoption down?

It does for me. But probably not for a lot of folks. Most people dont care.

What does speed adoption is to have a very well designed command line interface. Here go and npm excel.

There is basically only one person working on LuaRocks. He is also a PhD student and the maintainer of htop. Given this I think anything is "excusable".

Package signing is an important feature to have but it requires resources to set up. If you want to help: http://luarocks.org/en/Mailing_list :)

Wow, didn't know he also authored htop.

After visiting his site[1], discovered gobolinux[2] too!

1. http://hisham.hm/

2. http://www.gobolinux.org/

LuaRocks dev here, thanks for the feedback!

> But its biggest crime is that packages are neither signed nor are they transported over https.

Do you mean the actual LuaRocks source code tarballs?

If you need to download the source code over https, you can fetch it from Github. We could mirror the tarballs there too, if there's interest.

> That said, some of moonrocks goes over http, https://rocks.moonscript.org/

Rocks nowadays are transported by the `luarocks` tool over https by default. There are also http-only mirrors.

Oh wow!

It blows my mind that the author of htop and gobolinux and luarocks are the same person. Much respect. I use htop every day. And Nix and GoboLinux are architectural inspirations.

What I am saying about luarocks is that all code pulled over the wire should be signed and or encrypted to maintain the chain of trust. Maven just went through this, as it was only accessible over HTTP unless users paid a fee.

http://www.infoq.com/news/2014/08/Maven-SSL-Default

I could setup a MITM on a local wifi network (hacker space, coffee shop, school network, etc) and installed backdoors via luarocks because the traffic goes in the clear and the packages are not signed.

Luarocks should be secure by default. I found it really hard to activate SSL in luarocks.

How does LuaRocks implement the cross-compilation? Lua is widely used on embedded platforms, so I wonder how easy it will be to just take a random module from the LuaRocks and build/deploy it for some MIPS or ARM board?

Pure Lua modules do not need to be cross-compiled.

Simple C-based modules using the "builting" build back-end will probably work well with cross-compilation just by setting a few variables (CC, LD, CFLAGS...).

More complex modules will use the "make" or "cmake" back-ends and you will need to pass specific options to their build systems, which LuaRocks can do.

See http://luarocks.org/en/Config_file_format for more information.

Thanks! Config file looks like a powerful way to customize builds. But am I right saying that I will have two configs (for host and target) and I will have to pass --force-config option to toggle them?

No, you can use environment variables to choose at runtime (first paragraph of http://luarocks.org/en/Config_file_format)

Reply on news.ycombinator.com