Hacker News Reader

Comment by lmm | Hacker News Reader

Why Lua rather than e.g. TCL or Python (both similarly easy to embed, but better-known as full-fledged languages)?

TCL's good for string stuff, but gets very messy if you want to do stuff outside of that. It wasn't really designed originally as a general language: it works and does have some (IMO 'too') clever features, but it has a lot of foot guns as well: comments are actually (almost) ignored procedures, which causes issues, you sometimes have to escape comments or they'll change logic or cause syntax issues (i.e. trying to comment out statements sometimes still triggers syntax errors within the comment!), everything's a string which is great for strings, but not when you need to start validating numbers or similar... TCL makes a lot more sense as a command language (what it was originally designed for - string commands) or a REPL...

Back in the late 80s before Python and Lua were released the following decades, TCL made sense, as it was the only freely available embedable language.

Python's larger and more complete (and I'd argue a better language then Lua), but Lua's compact and very fast as it's a register-based bytecode VM (and luaJIT exists which is even faster) (although if you don't use the 'local' keyword on variables, it's then quite a bit slower as it no longer uses stack-based variables, so the code can end up being more verbose to make it fast), so games commonly used Lua for scripting/gameplay as it was easy to integrate.

Have you seen TCL quadcode? It infers types and uses llvm - Check out the typing diagram; it is kind of crazy. Maybe it is to avoid shimmering more than anything, but I think it is impressive. Thought you might find it interestung even if you don't use TCL.

https://wiki.tcl-lang.org/page/tclquadcode

Let me add my voice to the chorus of posters pointing out that, no, Python is not easy to embed.

Back in 2004 or 2005, Firaxis decided to use Python as their embedded scripting language. They used something called “Boost Python”, a then reasonably easy to embed fork of Python, to embed Python2 in their Civilization 4 gaming engine.

Soon after this, Boost Python got abandoned and Firaxis ended up having to use an outdated version of Python by the time they released their final Civilization 4 expansion.

For Civilization 5, Firaxis instead used Lua, since they wanted an actively maintained code base.

For my own “embed a scripting language in a DNS server” project, I went with a slightly modified Lua 5.1. The entire DNS server, including the Lua scripting engine, is a 103,936 byte sized Windows service. The stack was a little hard to grok at first, but I was able to fairly quickly get used to it and have a Lua script set up configuration for the server, as well as parse DNS queries. [1]

To Python’s credit, the Python2 code used in Civilization 4 is 100% compatible with the final 2019 release of a Python2 interpreter, to the point that I can run map scripts for Civilization 4 -- compiled for x86/32-bit -- on a 64-bit ARM Raspberry Pi and have them generate the exact same maps. Useful when I wanted a particular kind of map for a Civ4 mod, and had to iterate through 300 different random seeds on my Raspberry Pi to find the desired kind of map. After about a month, I had over 180 map seeds meeting my criteria.

[1] https://github.com/samboy/MaraDNS/tree/master/deadwood-githu... for the record

Some corrections: Boost.Python (note the period) was a C++ library for automaticing the generation of cross-language bindings between Python and C++. It wasn't itself a distribution of Python. Boost.Python still exists. However, its successor (pybind11) used features shipped in C++11 to simplify the implementation (and compiles far faster) and is the leading Python-to-C++ binding to use today. I'm using pybind11 in a project today with bog standard Python3 and it works great.

Lua made inroads into the game dev community in particular thanks to LuaJIT. Despite it actually being a fork of Lua that hasn't kept up with changes in the base language, LuaJIT remains popular for its speed.

Python is all but "easy to embed" - for one thing it's huge, and it requires linking to native libraries which may conflict with your own (openssl comes to mind). It also has an unstable ABI (and even API in some cases). Lua, in contrast, is just a single, very lightweight DLL, with no external dependencies and a stable ABI.

Yeah, Python is "theoretically" easy to embed, that's why it uses the GIL

In practice, it is hard to embed and we get the GIL disadvantages, so that is not working out so fine.

The GIL is less about python being easy to embed, than it is about embedding other libraries into python. The GIL was set up to make it easy to wrap C code and expose it to python code. A task for which it served quite well at the time and to be honest lasted longer than I would have expected.

My impression is that Python is not actually easy to embed (and I say this as a fan and frequent user of Python).

Modern tcl is big. Embedding something like Jim is still possible, but I think Lua is more popular than tcl and has a more approachable syntax if you want laymen to use your DSL.

Modern Tcl may be big, but it is not too big to embed. I know, because I embedded it into a Go system at a previous employer. Unfortunately, it is proprietary, so I may not point you to the repo, but it required remarkably little glue code.

I also evaluated Python and Lua. As others have noted, Python appears to be a right royal pain to embed. I actually had more experience with Lua prior to that project, but that experience leads me to believe that Lua is, generally, the wrong language.

Looking back on the project, I would not hesitate to do the same thing again, ideally as open source so that other Go projects can embed a scripting language. It got the job done, and did it well. Non-programmers on our team were able to write both configuration and logic successfully & productively, and apparently enjoyed the experience. Tcl itself ended up being a pleasure to use. While I personally would have enjoyed something like Embeddable Common Lisp, I think that would have been to much to ask of the rest of the team.

Interestingly, the four languages Lua, Python, Tcl and Lisp each can be said to take one idea and run with it. Lua is everything-is-a-hash-table (well, almost everything); Python is everything-is-an-object; Tcl is everything-is-a-string; and Lisp is everything-is-a-list (well, in theory: in practice it is really everything-is-an-object). I don’t know if this says anything deep about scripting languages, but it is at least interesting — right?

Python is not easy to embed. It's very unsecure and is difficult, if not nigh-on impossible, to effectively sandbox. (see RestrictedPython)

It's my understanding that Lua can be more easely sandboxed, at least compared to Python.

Lua is smaller, and one of the fastest in the bytecode-land. It also used in a large number of diverse applications [1], so in terms of popularity, I think it has nothing to be ashamed of compared to others.

[1] https://en.wikipedia.org/wiki/List_of_applications_using_Lua

When I say "embed" I mean: add the source tree to your project and compile statically (see the CMake project for Lua 5.4).

To my knowledge, you can't do that with Python. I don't know about TCL though.

Not sure about TCL, but python is much bigger, has a bigger memory footprint, and is a bit slower than LUA. Also, check how easy is to integrate Lua into your codebase

I embedded Python and it was not easy to embed at all.

Reply on news.ycombinator.com