Hacker News Reader

Comment by pansa2 | Hacker News Reader

One obvious improvement here is the use of `HPyContext`. This allows the Python interpreter state to be encapsulated rather than globally-accessible, which would enable multiple interpreters in a single process.

Lua’s C API is generally considered to be well-designed, and does something similar with `lua_State`. How does the HPy API compare to the Lua API more generally?

View on HN

> enable multiple interpreters in a single process.

CPython (the default interpreter) has had support for multiple interpreters in the same process since 1997, but it has only been exposed to the C API, and not in the language itself. Python 3.10, coming out later this year, will expose multiple interpreters in one process (subinterpreters, see PEP 554 [1]).

I'm excited about what could eventually come out of this. If there is one GIL per interpreter, we could have something like the `multiprocessing` library for parallel execution, but all within one process.

1 https://www.python.org/dev/peps/pep-0554/

This is basically how the Threads extension [0] works for Tcl. Tcl has long supported creating many interpreters per thread. The Thread extension exposes some threading commands to the interpreter, which lets you create a new thread+interp. You can then send messages to that interpreter, or pass it file descriptors, but otherwise they are fairly isolated (apartment threading model).

[0] https://www.tcl-lang.org/man/tcl8.6/ThreadCmd/thread.htm

Can multiple Tcl interpreters run truly parallel on different threads in the same process? That’s possible with Lua but AFAIK not with Python, because one of the pieces of global Python interpreter state is the Global Interpreter Lock.

Encapsulated interpreter state via HPyContext would allow replacing the per-process lock with a per-interpreter lock, enabling such parallelism.

Yes, threads from the Thread extension are real OS threads, which have a Tcl interp in them. Since they are separate interpreters then there's no impact from any kind of "per-interpreter lock".

Passing messages between Threads can be done asynchronously which does not block either thread.

Example Tcl program with threads

    package require Thread
    set tid [thread::create]
    thread::send -async $tid {
        while true { after 200; puts Jazz }
    }
    while true { after 100; puts Party }

This creates one thread which is in an infinite loop printing "Jazz" to stdout every 200ms, and the "main" thread which is in an infinite loop printing "Party" to stdout every 100ms.

Related but separate from the subinterpreters PEP, there is interest in moving from a global interpreter lock to one lock per subinterpreter. This would need to be in place before running Python interpreters truly in parallel within one process. There’s a great summary here: https://lwn.net/Articles/820424/

No. See A Disclaimer about the GIL right after the Abstract.

The big difference between Lua and Python C APIs is that Python values are exposed directly as first-class values (as pointers in CPython, and as handles in HPy). If one native function wants to call another native function and pass it a Python object, it can do so directly - the VM is none the wiser for it, except for reference count updates.

In Lua API, all Lua values are bound to a stack, and C functions have to exchange values via that stack - there's no free-standing "Lua value" type in the API. Thus, the runtime is fully aware of all the objects that flow back and forth, even if one native Lua function is calling another native Lua function.

> In Lua API, all Lua values are bound to a stack, and C functions have to exchange values via that stack - there's no free-standing "Lua value" type in the API.

That's not 100% correct. You can have values unbound from the stack, they're just typed, rather than having one "value" type. (lua_Number, lua_CFunction, etc). Though functions bound from do still use the stack for taking/returning values. (However your C functions can directly work on other value types).

Correct me if I'm wrong, but doesn't this only work for value types? i.e. there's no way you can hold a reference to a Lua (not C) function, or a table, without the stack.

What you're looking for in that case is lua_topointer, which can turn any Lua userdata, table, thread, or function into a void pointer.

Not really. Looking at the doc for that function:

"There is no way to convert the pointer back to its original value. Typically this function is used only for debug information."

(In CPython, you can e.g. stash a PyObject* away in C globals.)

If what you want is a reference to any Lua object, that you can grab from somewhere random in C, that doesn't exist on the stack, then you'll be using Lua's registry.

Your first stop will be luaL_ref [0].

[0] https://www.lua.org/manual/5.3/manual.html#luaL_ref

Yep, that's exactly what I meant. It goes back to the same thing - all Lua objects exist strictly in the "Lua world" (be it the stack or the registry), and anything that wants to exchange them has to go through those mechanisms. In Python, you just get a PyObject*, and you can pass it around as much as you want - the only thing you have to take care of is to reference-count it properly.

A registry object in Lua is an integer in C, and you can pass it around as much as you want. However, you have to be careful, because it can be eliminated by the GC, in much the same way as a PyObject pointer.

The registry object really is just an address. It's the equivalent of: lua_State->top+LUA_REGISTRYINDEX[reference] in C.

There's no functional difference to a PyObject pointer.

Reply on news.ycombinator.com