When FFI function calls beat native C

No, I don't think Lua (or the author) is inlining the external function.

What's being JIT'd is the Lua code into machine code, and where Lua would need to call a C function, it (apparently) is just emitting a `call the_external_fn` into the JIT's resulting assembly. That's a direct function call, so it's about as fast as you're going to get, but somewhat counter-intuitively, it'll be faster than C, as we don't do anything with the PLT or indirection. Just call the function.

How does it run with the new FFI?

You could export a varargs function and use this to call the appropriate Lua function, I guess.

Lua (and therefore LuaJIT) can easily be driven from a C program, if that's what you mean.

LuaJIT also allows you to pass a Lua function as a callback to a C function expecting one, subject to some limitations.

I'm not sure if either of those answers your question though.

Sorry for the late reply, lost the tab for a bit ._.

Something that might be relevant is that that LuaJIT can JIT-optimize FFI calls from Lua but can't/doesn't optimise calls into Lua made via C.

I might be able to go digging for the reference (at this point figure it's best to just reply for now, not sure if you'll see this) but I've read that "the approach" recommended to solve this problem is to move the main `for(;;)` / `while(1)` loop into Lua and have LuaJIT repeatedly FFI-call C, because that's the path that can go the fastest.

The better takeaway is that on ELF/PIC platforms a call to dynamically linked function is somewhat more expensive than one would assume.

With modern CPU architectures that can prefetch/speculate over indirect jump/call making it comparatively expensive to indirect call in a tight loop (the idea behind PLT is that in contrast to indirect call through GOT it should more clearly signal the intent to the CPU). On the other hand the tight loop is certainly important and this effect will not be so pronounced on any kind of practical code (because of the BTB pressure).

Also, this is interesting observation for various discussions about overhead of late binding ("virtual" in C++) as similar overhead is already there for almost any cross-object function call in PIC dynamic binary.

ELF relocations can also patch the machine code if I understand correctly.

This is about different tradeoffs. On typical unix ELF platform the idea is to preserve memory by sharing as much of _shared_ object code as possible, while NT emphasizes the _dynamic_ linking. Notably the approach of just giving up, unsharing and relinking the PE/COFF when it does not fit at its preferred load address is not the only one implemented by windows variants: Windows 9x is designed for memory starved environments and actually implements the PE/COFF linker in kernel and can relink on per-page basis when swapping the page in (W9x is stupidly complex ridiculously brilliant hack that solves weird problems in wonderful ways) and most builds of Windows CE will just error out when relinking would be required. In fact Windows SDK contains tool that relinks some set of PE/COFF DLLs such as to minimize load conflicts (for NT/9x it is performance optimalization, for CE it is often necessary).

On typical ELF platform with dynamic linking you get PIC compiled binaries with the added overhead of GOT and PLT. One thing to keep in mind is that on many traditional unix RISC platforms there is something similar to GOT and PLT even in statically linked binaries (no word-sized immediaties on RISC platforms…).

The article has benchmark results that quantifiably establish direct calls outperforming indirect calls.

RET is typically special cased in the speculation logic. I suspect that this construction will have significantly larger miss prediction rate (on some uArchs quite possibly 100%) than straight indirect jump.

The answer is in the article

> The downside to this approach is slower loading, larger binaries, and less sharing of code pages between different processes. It’s slower loading because every dynamic call site needs to be patched before the program can begin execution. The binary is larger because each of these call sites needs an entry in the relocation table. And the lack of sharing is due to the code pages being modified.

It probably could be improved by essentially doing the same thing that luajit is doing and inventing a JIT code loading mechanism, but that would be hard to get a lot of buy-in. You need to change the ELF standard and get the likes of GCC and LLVM on board with this new paradigm.

there could also be problems on some architectures. on arm64 the branch-link instruction can only jump +/- 128MB (https://developer.arm.com/documentation/dui0802/a/A64-Genera...). so if you want to use BL instead of BLR for performance reasons then you need to make sure all the shared libraries are loaded in the same 128MB address space.

gcc has an option to reduce the overhead with no-plt (https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html). so on x86_64 it just does call[rip + got_offset]. this is still an indirect call but it reduces the number of calls by 1.

Because nobody cares about performance. As far as I remember on Linux every C symbol not explicitly inline or static defaults to visible and dynamically linked. You can try to tell gcc to treat every symbol as hidden by default but at that point every third party header would have to explicitly declare its exported symbols as visible. That whole dllexport/import mess Windows binaries require? That is the implicit default on Linux.

yes. my jit which does the same thing was 10x faster on windows than on Linux. apparently the elf hash lookup beats the linear or whatever primitive search on coff.