Creating C closures from Lua closures | Hacker News Reader

Creating C closures from Lua closures

lowkpro.com

60 pointspublicdebates16 comments

This doesn't seem like something that should require generating assembly to solve. Couldn't `CALLBACK` just return a table, which contains both the userdata pointer to `REAL_CALLBACK` and a value for `findex`, eliminating the global variable? Then `Add` could extract that. You could even make the table returned by `CALLBACK` callable in Lua with a metatable.

Or, if you're worried about performance/memory, you could allocate a struct for `CALLBACK` to return as userdata, containing the function pointer and `findex`. If you made a little allocator for these it would be extremely fast.

I'm sure I'm missing things, but the solution you chose feels like the nuclear option.

The code is passing the function pointers into Win32 APIs, so the caller side isn’t controlled; the callbacks have to work as native C function pointers.

This was probably posted in response to this other link from two days ago, which is about about JIT-compiling wndproc callbacks in particular; the comments discuss the “proper” way to do it, which is to use GWLP_USERDATA:

https://news.ycombinator.com/item?id=46259334

At least, that’s the proper way to do it if you control the entire application. But for what’s apparently a thin wrapper that exposes Win32 APIs directly to Lua, I can understand picking an approach that makes life easier for the Lua code author, even if it’s hackier under the hood. It also avoids the need to write custom behavior for each API.

It was inspired by that post, but not in response to it. Over the 3 or 4 months that I took writing lowkPRO, one of the features I'm most proud of is this, the ability for it to create C callbacks on the fly that just call Lua callbacks and return the value to C. And I would argue that it's not hacky but just well-engineered. After all, everything compiles to assembly at the end of the day, right?

Well designed C APIs have a context/userdata parameter on their callbacks, which is registered and stored alongside the function pointer. Unfortunately WNDPROC lacks this parameter.

GWLP_USERDATA should be the best option, though the API for setting it and setting the WNDPROC being separate looks error prone.

The bigger problem is that I would have to automate this for every callback type in the Windows API, and there's no guarantee that all of them follow the same 2 or 3 patterns for passing the context pointer into the callback. This solution works great, even if it is a bit wasteful of the ctx ptr.

I feel libffi is better suited and supports most platforms.

Tsoding made a video about it : https://www.youtube.com/watch?v=0o8Ex8mXigU

The ffcall library even more

wasn't aware of ffcall, it's definitely a better fit

Fun fact: LuaJIT FFI actually has a similar feature. You can do funky things like detour hooking with this functionality too.

https://luajit.org/ext_ffi_semantics.html

What’s the thing emulating so much activity around closures lately?

https://news.ycombinator.com/item?id=46228597

https://news.ycombinator.com/item?id=46259334

Also talking about the Knuth boy/man test: https://news.ycombinator.com/item?id=46020151

Not a bad thing, but that really question if there is some active micro-

Why VirtualAlloc?

Lua has its own allocator, which will also collect for you. lua_newuserdata. At the expense of having to set executable yourself, but without all the inbuilt inefficiencies the article points out.

Can you set arbitrary memory allocated by malloc (e.g. HeapAlloc in Windows) to executable, via VirtualProtect or something else? If so, that's news to me. I thought it had to be memory allocated by VirtualAlloc only.

That said, I'm not sure that solution beats out this one. I'm using a linked list on top of an arena allocator in practice, which means allocations happen once every 0x10000 bytes. (A single C closure of mine takes up exactly 16 bytes, which includes the pointer to the next linked list node. I'm very happy with how it's designed.)

Lua lets you set the allocator, so you don't have to manage the lifetime.

    void lua_setallocf (lua_State *L, lua_Alloc f, void *ud);

Two problems with that.

First, it would mean that all Lua objects would be allocated in memory that's potentially executable. That might not necessarily be more of a security issue than the app inherently allows, but at least it's wasteful.

Second, I'm not entirely sure it's a good idea to have C callbacks be collectable, which is why currently they aren't, and you have to free them manually via my winapi.freecallback function. This is part of the inherent issue in the fact that the Windows API has its own concept of lifetimes that are different than Lua's collection visibility. I would like to try to tie them together if possible one day, I'm just not entirely sure how yet.

I wonder if there would be a way to piggyback on top of GCC's nested function extension. It does something a little bit similar, with the dynamically generated functions.

Reply on news.ycombinator.com