Try building (the `v2.1` branch of) LuaJIT with `CFLAGS=-DLUAJIT_ENABLE_GC64`. This flag is a relatively recent addition [1], and didn't have JIT support on x64 until really very recently [2], but it should free you from the 4gb limitation.

[1] https://github.com/LuaJIT/LuaJIT/commit/a92e73023353e59405eb... [2] https://github.com/LuaJIT/LuaJIT/commit/2868715d80b6ac497a7f...