Bytecode Breakdown: Unraveling Factorio's Lua Security Flaws | Hacker News Reader

Bytecode Breakdown: Unraveling Factorio's Lua Security Flaws

memorycorruption.net

466 pointsmemcorruption122 comments

Unexpected!

Since lua interprets bytecode, it can check the arguments to the bytecode are meaningful. Point to memory lua allocated, things like that.

Turns out it doesn't do that. Feed it bytecode with invalid arguments passed to the instructions and it executes it anyway. The rest of the compromise follows.

Further, instead of fixing their interpreter, the game plan is to statically analyse bytecode. Which turns out to only work in simple cases.

For a sandbox friendly interpreted language this is pretty disappointing. I wonder if they'd take patches to stop the interpreter trusting the input - presumably performance regression is the fear there, which seems dubious when luajit is the fast option anyway.

Sandbox friendly?

It's not exactly a simple task to make safe interpreter for bytecode as some other languages have shown. It's a trade-off which also simplifies the reference implementation.

I wouldn't trust most of these interpreters when it comes to third party code execution, I barely trust my browser even with all the R&D money and attention web browsers receive.

Sandboxed in the sense that things like file i/o or network access can be easily removed and selectively reintroduced, e.g. to give an interpreter which can trash it's own heap but can't do anything to the host.

Bounds checking on instruction opcodes can absolutely be implemented in an interpreter. I suppose it's more complicated than just trusting the integer - but then the thing doesn't fall over on malformed bytecode which seems like a feature.

> I wonder if they'd take patches to stop the interpreter trusting the input

The stance of the Lua developers AIUI is that processes that execute arbitrary Lua code should only accept source, and disable direct loading of byte code.

This seems reasonable to me, as it preserves the option of directly loading trusted byte code, while avoiding putting dynamic checks into the interpreter that would affect all users.

Correct. From the Lua 5.2 manual [0]:

"all functions that load code are potentially insecure when loading untrusted binary data. [...] When in doubt, use the mode argument of those functions to restrict them to loading textual chunks."

I thought this was widely known within the Lua community - I'm amazed that Factorio would support loading untrusted bytecode. Was that a decision the game's developers actively made (for what reasons?) or were they simply unaware of the risks?

[0] https://www.lua.org/manual/5.2/manual.html#8.2

Java, Wasm and BPF demonstrate that it is possible to have statically-verifiable bytecode for JIT-compiled languages. Lua’s problem is that the bytecode doesn’t provide the information necessary to fully verify its safety.

All of those formats are designed to be translated to machine code when maximum performance is desired. Whereas Lua byte code is designed and optimized to be interpreted directly.

One step in Lua's evolution was to change from a stack machine to a register machine: https://www.lua.org/doc/jucs05.pdf This made the interpreter faster, but also (I suspect) more difficult to verify. I believe both Java and Wasm are stack machines (don't know about BPF).

Java was not. It was originally designed to be interpreted, and is still interpreted in many implementations (especially before any JIT kicks in).

In any case, those were just the examples I could think of OTOH. MSIL/CIL is another good example; I’m sure there are many others.

Technically speaking, CIL is always compiled by CoreCLR (it has interpreter internally but it is never used and therefore has succumbed to bitrot), it is sometimes interpreted by Mono on certain platforms as a stand in for dynamically emitted code. A special case used to be with WASM target with Mono but supposedly that's in the past.

I wonder if this affects Roblox variant of Lua?

Luau (Roblox's variant of Lua) seems to have disabled loading bytecode from Lua completely. Per https://luau-lang.org/sandbox:

> To achieve memory safety, access to function bytecode has been removed. Bytecode is hard to validate and using untrusted bytecode may lead to exploits. Thus, loadstring doesn’t work with bytecode inputs, and string.dump/load have been removed as they aren’t necessary anymore. When embedding Luau, bytecode should be encrypted/signed to prevent MITM attacks as well, as the VM assumes that the bytecode was generated by the Luau compiler (which never produces invalid/unsafe bytecode).

>> the VM assumes that the bytecode was generated by the Luau compiler (which never produces invalid/unsafe bytecode)

Yep, to that end they also have a basic bytecode verifier (only used in debug mode / when asserts are enabled) that validates the compiler only outputs valid bytecode, and I believe they continuously fuzz the compiler to make sure those asserts can't be triggered. See https://github.com/luau-lang/luau/blob/0d2688844ab285af1ef52...

It's fairly robust (and Luau bytecode isn't _that_ complex,) but they made the right decision disallowing direct bytecode execution.

No, expected. Only execute bytecode that was actually generated by the correct compiler. Otherwise you get memory safety violations or sandbox escapes. (Or sandbox escapes via memory safety violations.)

Just as you don't execute arbitrary machine code.

Luau has the same thing, and you don't see Roblox suffering from sandbox escapes all the time, do you?

Lua isn't really sandbox friendly, although that's a common misconception.

Lua (by design) doesn't provide termination guarantees or a good way to force an untrusted program to terminate. If you accept untrusted lua input, be prepared for your program to halt indefinitely.

Lua is great for semi-trusted input, AKA things you download from the internet, where you do a minimal amount of due diligence, but in case the code is actually malicious, you want to severely limit (but not completely eliminate) the harm it can do.

If you actually need Javascript-style completely untrusted input,, what you want is the Roblox fork called Luau[1].

[1] https://luau-lang.org/sandbox

Any language can be sandboxed on the VM level. It's a property of it's implementation. So you can say that Lua has no sandbox friendly implementation right now.

For example, termination you can solve by unwinding the stack in efficiently polled safepoints. You need to take down the entire sandbox-capable Lua VM instance but you can.

Lua has debug hooks that can be used for the purpose; for example an instruction counter coupled with a pool allocator should get one quite far. I would never trust third party bytecode, only source code.

Is anyone familiar with Roblox luau security features?

Luau can be sandboxed, and scripts limited in cpu and memory usage.

I wish this was more defined or documented somehow. You're kind of left on your own to figure out whether a language is reasonably guaranteed to be safe or not.

Some example scenarios:

- Code is static and is executed directly by a user, the default case languages care typically care about. Including Lua.

- Code is dynamically fetched and executed through some update process, hopefully only through official channels. Here you can get away by making the process secure, but who knows.

- Code can be added by the user through plugins, this can be made easier through stores with the click of a button. You can review plugins, but this is hardly done. Here you need to consider if the code should be sandboxed or the user should be careful.

- A multiplayer game where a server can be extended with custom code via plugins, but not the clients. Here you need to consider that the users/gamers who are hosting servers are eager to try many different plugins. The plugin community (gamers) can also be a lot more dangerous.

- A multiplayer game where the server can execute arbitrary code on clients, just like a browser. Here you need to be very careful about sandboxing, especially on clients as gamers will just join random servers without thinking about the security implications.

The last point being Factorio's case. I'm not necessarily disagreeing that it's the developers job to evaluate this, but sometimes it's not obvious that for example the load function in Lua can run arbitrary bytecode which is unsafe.

To be honest, I wasn't aware that Lua's bytecode is unsafe, but I am aware that LuaJIT's bytecode is unsafe. But as far as I know this fact is just stated randomly in the mailing list and github issues as an obvious fact.

There is another thing about servers being able to crash clients (just run some infinite loop on them), but this much harder and maybe pointless to avoid.

> A multiplayer game where a server can be extended with custom code via plugins

A game called Mordhau (based on Unreal engine) had a built-in "message of the day" feature where server owners can put in a URL that loads an in-game browser when the player connects. No client side option existed to disable the browser and I believe the devs eventually disabled it completely but I'm not sure the status of it now.

Just shows how complex games / game engines are getting where you have an embedded web browser for seemingly no good reason.

A lot of games have web browsers embedded nowadays for ui

Unreal Engine has something that can be called embedded web browser since the day one. The original Unreal Engine is this thing that has its own implementation of “something not entirely unlike JVM” and refers to various things by means of URLs.

The game "Tabletop Simulator" allows you to spawn various objects into a VR playspace. One of the objects is a tablet PC, which displays a little web browser on its virtual screen. It's handy to look up rules or whatever without leaving VR.

The last time I tried this, the browser had a notification that it was out of date and needed updates.

Thinking about this broke my brain a little. I have no idea how to apply software updates to the virtual browser on the virtual tablet running in a virtual room simulated by my (hopefully real) PC.

Garry's mod uses Lua on server and clients. It also has the ability to create an embedded web browser on the client.

There have been many exploits throughout the years, including this particular exploit with bytecode, though in LuaJIT. Some were source engine related, some LuaJIT related, some web browser related (Awesomium) and some even steam overlay related.

I believe one funny thing about Awesomium was being able to read arbitrary files outside of the sandboxed virtual file system by using the file:// uri scheme. I think some debug related commands in source engine would also allow you to get a list of files outside of the virtual file system.

At one point someone even managed to install actual malware on my computer and sent me screenshots of my desktop. I forgot what the exploit was though.

> sent me screenshots of my desktop

Damn. That's the scariest thing I've read all week.

This thread is really making me consider buying another computer for all gaming related things...

The first thing to look for is if the solution states clearly that it is a speculation-safe sandbox. I do think that not many will do that, but there are some. And go from there.

You should never assume any method of executing any attacker controlled code is safe, unless something explicitly calls that out and also has put Google-level amounts of effort into supporting that.

My interpreter only accepts print and addition to a predefined variable. Let the attackers print and count all they want.

The problem isn’t the execution, it’s the scope of what it means to “execute”.

Depending on the implementation, there might still be multiple bugs lurking, especially in input parsing

Factorio has a really good dev team behind it so I trust that they're doing their best to fix these issues. Though, gamedev in general seems to be more of a creative endeavor which puts things like code practices and security on the back burner. I wonder how many zero day exploits are lurking in game clients / servers.

Most likely, it’s not very good. Why do you think every console manufacturer, from Xbox to Sony to Nintendo, does not allow connecting to arbitrary server IP addresses or modding support?

It’s not merely a business decision (like some believe) to force people to use official Online services. Think about it: Restricting connecting to third-party server IPs means that any bugs in the network code, or in the rest of the game, even atrocious ones, will never be exploited. Restricting mods (even “safe” mods like Lua) further prevents exploits. This makes sense - buggy network code has tanked multiple consoles’s DRM historically.

And not just exploits - these consoles pride themselves on doing their review process before the code becomes available (despite oversights). Allowing executing of Lua, from a remote system, basically means a game could be reconfigured remotely after approval potentially from the developer themselves - not something any console manufacturer wants to permit without very close inspection.

I don't own any newer console, but I remember people abusing the P2P nature of PS3 Call of Duty MW2 to make ridiculous custom servers that you would randomly connect to. So at least on the Playstation 3, games would connect to arbitrary IPs. Maybe this has changed in the PS4 and PS5, probably for this reason.

If there's a specific host/server that everyone in that game has to connect to, it isn't "P2P".

Pedantically, maybe. But the host of the match lobby was randomly selected and it could have been any of the peers who end up as the host. When the host leaves, one of the remaining peers becomes the new host. So, all peers have the capability of being client and/or server.

That sounds like p2p to me.

Ah, I didn't know something like this existed, I thought you were talking about player-hosted lobbies.

You connect directly to one of the players in the lobby, also known as a peer. It isn't a dedicated game server, just the game itself. I think it qualifies very well as P2P.

I remember playing the Xbox version and you would sometimes get lobbies where everyone had infinite ammo grenade launchers with no reload, or other similar funny stuff. I really enjoyed myself when I was lucky to end up in one of the matches.

Yeah, best to keep a separate computer for gaming for this reason. Definitely don’t put important documents or work stuff on it. It would be ideal to isolate it in a VM, but setting up a gaming VM can be a massive pain in the ass and exclude you from some games that use anticheat.

Yup. Intend to assume that any game with remote interaction is completely insecure. It is best to run Steam and all games in some sort of sandbox.

Flatpak is probably a helpful start. While containers aren't a strong security boundary at least simple exploits won't work.

Code practices? Factorio is one of the most well programmed, stable, and consistent piece of software I've ever seen. It's almost a shame to see skilled people work in games because of how desperately other fields need people who are good at programming.

See my comment: > Factorio has a really good dev team

I wasn't talking about the Factorio devs specifically but about the gamedev industry as a whole.

In general, verifying programs is extremely hard, not just because of rice's theorem but because it's so easy to miss a spot, especially for non-trivial bytecode languages like lua's. wasm has no concepts of for loops for example.

It's strange that after upstream has given up on the problem as it was too hard, factorio devs have chosen to try to fix the verifier/write their own (not sure which of the two they did).

Minetest's loadstring function forbids bytecode entirely: https://github.com/minetest/minetest/blob/9a1501ae89ffe79c38...

I wonder why factorio mods need the ability to execute raw lua bytecode. If they don't have it, there would be no need for a verifier.

It's quite dangerous in the first place to execute lua code downloaded over the network. JS execution environments have gone through decades of cycles of discoveries of exploits and fixes. Lua gets those as well but on a smaller scale, and with less staffing to improve security. The main protection is I guess that there is fewer people running malicious game servers.

Factorio has stuff like this:

https://mods.factorio.com/mod/Moon_Logic

Besides, it's quite limiting to create software that can't just execute in a Turing complete environment.

Anyway, we really need interpreters that include a strong capability system.

“If I ran the business” (TM), I would just put it in Factorio settings as a toggle switch called “Reduced Security Mode - Allow Lua Bytecode.” By default, it’s turned off. People who really want those mods can turn it on, as long as they are informed (UAC prompt style) that they better trust the mod authors. I’d also add an API for mod authors to detect if bytecode access is enabled; so they can make their mods compatible with either environment.

Or maybe, down the road, Factorio could enable mods with greater privileges, as long as they are open source, and do an App Store-style review process with the community. Approved mods get not just bytecode, but perhaps even some of the typically forbidden modules like filesystem access. Unapproved mods using those enhanced privileges won’t run without a startup flag.

> By default, it’s turned off. People who really want those mods can turn it on

This works until a popular mod requires it (for legit reasons) then enabling the option becomes the de-facto standard for people who install mods.

It's possible to make it so you enable it on a per mod basis, like app permissions

Because that's never ever been a total usability disaster that just encourages people to enable every one because they don't feel like fighting it. It's also not effective, given how tightly tied mods are to the core API.

When you're talking about security, adding a bunch of config flags for users is never a good idea. Most users aren't going to understand what it does, and like others mentioned, there's too many reasons to turn it on.

I see no problem. It's those people's choice to do so, and they accept the risk.

OP only refers to bytecode. There's nothing wrong with executing Lua when provided to the VM via source code. The only reason to allow the VM to load bytecode directly is 1) a very minor improvement in loading time, as the interpreter then doesn't have to parse Lua code into bytecode itself 2) allowing obfuscation of logic running within Factorio. Both seem rather irrelevant, so I'm not sure why they allow loading bytecode directly.

Isn't hardening source code parsing much more difficult then hardening byte code parsing?

Sure, the output of the lua compiler might be guaranteed to not invoke undefined behavior in the byte code interpreter, but the compiler itself might be vulnerable.

To my knowledge, compilers like GCC and clang treat source code as trusted for this reason

Factorio disabled bytecode loading in response to this. Bytecode did allow for some cool stuff like writing mods in a preprocessor language that spits out Lua bytecode, but ultimately the security issues were more important to address.

Almost all of the debug library was made unavailable to mods as well, for similar security reasons.

Citation?

Factorio 1.1.101 (which the blog post says included the fix) does not list any changes regarding the disabling of bytecode or restricting the debug library. This would have been notable news, even without admitting the security risk. Factorio 1.1.107 does mention disabling the debug library, but it doesn’t seem this article had anything to do with that.

I work on the game. The debug library was disabled for other security holes that were brought to our attention, so it wouldn‘t be related to this, but I thought it was interesting to mention.

I believe the change was not mentioned in the changelog as an attempt at 'security through obscurity', trying to avoid people getting any ideas before the update is wide-spread. Not sure that helps any, but still.

Sorry, but thats just a perfect example why 'security through obscurity' is wrong. I have zero idea about security risks, but if fix does not mentioned anywhere, then for people that use previous version there no rush to upgrade.

I don‘t disagree.

Arguments either way. Generic "security vulnerabilities addressed" in release notes is a nice balance.

> no rush to upgrade

I suspect the overwhelming majority of Factorio players are using Steam, which auto updates.

Factorio is special though, because it actively uses the beta version functionality in Steam to not only provide betas but also older stable versions. This allows the devs to move fast and break things.

I know I've held back my copy of Factorio due to some concern over changes in newer versions, preferring to letting the dust settle before upgrading to the latest stable version.

Due to the need for perfect synchronization all users need to be using the exact same version. Mods can also break between versions. It is therefore very common for public servers to stick on one version for extended periods of time. It is common for people to use the Steam "betas" functionality to pick an exact version or download an exact version from the Factorio website.

I would say that servers only tend to update when large features are released. So announcing a security vulnerability would likely push some servers to update.

Without metrics of some kind from Wube I guess we aren't likely to know for sure, but I doubt very much it is common to run old versions of the game on Steam. I bet you that most people are simply running on the latest version at all times. That solves the MP issue, and plenty of mods don't need to be updated for each game version.

Loading raw bytecode is known to be unsafe, and iirc that is mentioned in lua_load/luaL_load* documentation.

A preprocessor could spit out Lua code with the same effect and less complexity. Really interesting why and how these decision were made.

For what it’s worth, Metalua also generated PUC-Lua bytecode directly instead of source code, making it incompatible with LuaJIT (which might have been part of the reason why it died).

Eventually every game developer learns the hard way that they must remove the bytecode ability from lua's loadstring() function.

E.g. here's a 12 year old blogpost on the topic from the ROBLOX developers: https://archive.is/oXPyM

To be honest, it would probably be better off disabled by default. Its legitimate uses are pretty niche.

Yep, its place is in luaconf.h really.

> not just because of rice's theorem

I don't think Rice is relevant at all. I guess Rice is a useful first screen. Do you believe you can "just" Decide this correctly? If so, Henry Rice got his PhD half a century ago for proving you can't do that, stop.

But assuming you've made your peace with accepting only a subset of the inputs that would actually meet your requirements, Rice is done. And you're right - now instead of an impossible task you've just got an extremely hard task. This means when you fail (which you will) at least nobody will tell you it was impossible, if that helps?

Factorio should not have done this.

This idea of accepting some kind of subset is exactly what JVM does. There is a set of rules (IIRC 29 of them) that the JVM bytecode have to follow to be accepted, the one important rule is “stack entries should always be used as a same type”, the rest of the rules are there so that this can be statically checked.

As I said, because of Rice accepting a subset is the only thing which could work. But it only could work, it probably won't because now you've gone from "Impossible" to merely "Incredibly hard" and that's not as a big a change as you'd hope.

> rice's theorem

That doesn't apply here. By the broad definition of "syntax" that Rice's theorem takes, the things you want to verify on the bytecode are syntax.

> In general, verifying programs is extremely hard, not just because of rice's theorem but because it's so easy to miss a spot, especially for non-trivial bytecode languages like lua's.

Perhaps a dumb question then. Java has famously had a bytecode verifier for decades. Is it the case that:

a) bytecode verification is fundamentally easier in statically typed languages

b) it's just as hard for Java, but Java has had decades to work on it and it's still taken a long time to fix all the bugs/security issues.

c) Java‘s VM uses a different architecture, and the bytecode is also designed differently.

Understood, but then what is it about the differences in the JVM and bytecode that make it more amenable to verification?

Total newb question but why do games use lua, as opposed to (for instance) embedded JavaScript with some defined interface into the game (e.g. APIs to adjust game state).

Seems to me that this would benefit from the much more severe hardening work that's gone into isolating browser environments (a hard and VERY well tested and funded target), as well as the massive work that's gone into performance optimizing dynamic types.

Plus if the mod needs UI there's the canvas and potentially even React etc if a Dom-like model is available.

It’s been several years since I’ve tried but most JS engines are out of date and barely maintained and the ones used for browsers are made for browsers first and they are not built for you to integrate at all.

Lua is specifically built for you to integrate so there are many resources and a huge community backing you.

That's helpful. Thanks for the response.

I think a useful caveat on that would be "...when these projects started". Roblox started in 2003 and Factorio in 2012.

As of today embedding v8 doesn't seem that hard (https://v8.dev/docs/embed)

1. Most JS engines are vastly more complex to embed than Lua, which is one of the easiest pieces of software to compile I can think of 2. You're conflating common browser APIs with JS. A JS engine does not provide a canvas, or a DOM. V8 does not, for example: those are things you would need to add yourself.

As a non-security dev, I'll drop the obligatory "wow this is incredibly impressive!" I can't believe how clear and logically you'd have to think to track down these intricate failure cases. Definitely not my strong suit! I'm much more of an "ideas guy" ;)

Content-wise: Wow... We are totally, completely, utterly screwed once people start putting together ensembles of AI SWEs equipped with 10,000 blog posts like this one on finding weird memory exploits. Ultimately I think we're gonna need a whole new paradigm for security, or at least some new element in the stack. It's my potentially naive opinion that all the modern talk about "trusted" clients and DB roles and all that is trying to patch holes in swiss cheese; hopefully, we can find a new stack of LLM-maintained swiss cheese to add on instead!

People are doing it. The results have yet to be promising.

So... this demonstrates an exploit that relies on a feature that is advertised as exploitable: loading byte code. What am I missing?

That advertised features can still cause harm to end users, particularly those who don’t know what Lua or bytecode are?

It’s possible that the bytecode interpreter has a bug that lets one run arbitrary bytecode, even in environments where `loadstring` is disabled.

The interesting takeaway I got was how badly the Lua developers failed on their bytecode veryfier. Not some complex issues, but simple ones like of by one errors when modelling basic instructions like jmp or the issue that the Lua interpreter would try to interpret anything it got its hands on as instructions, even data sections the veryfier would not touch.

> The interesting takeaway I got was how badly the Lua developers failed on their bytecode veryfier.

What verifier? The one they removed?

Or are you talking about the one the Factorio developers made, where flaws are a lot less surprising considering they have a lot less expertise with the internal machinery of Lua.

I seem to have skipped over the part where it mentioned that the JMP issue was in a factorio specific veryfier, despite reading the section several times.

“Even if the official bytecode verifier was not implemented in Lua 5.2.1, Factorio developers seem to have implemented their own in an attempt to protect the Lua interpreter”

We are seriously lucky such capable people are on the good side.

I guess it goes to show how many people are innately good or benign (not sure what word to use in English). The newsmedia would have you believe otherwise, and the average comments on such news perpetuate the belief, but if this were all true, how can we have all the luxuries we do? The healthcare and social aid programmes? It's not like there are no problems in the world, but clearly far more people are constructive than destructive

Perhaps this is too tangential but I guess it's front of mind since I just came from the HN thread about Panama papers where people were acting all unsurprised and like every wealthy person anywhere was evil and now got fully acquitted from any prosecution when neither is actually the case, as then a handful of comments pointed out (imo successfully, but you do have to read down the thread and not get caught up in cynicism)

IMO, Lua bytecode should never be usable anywhere outside of embedded systems that don't have enough resources to run the Lua source code parser. Besides security vulnerabilities, the only other thing it seems to be useful for is closed-source programs.

Unless I missed it (I admit I skimmed towards the end) The author does not discuss at all the actual remediation that was taken. I would love to hear more about that.

The remediation was:

1.1.104: https://github.com/Rseding91/Factorio-Lua/commit/4d924b69808...

and

1.1.107: https://github.com/Rseding91/Factorio-Lua/commit/ce12474c7fc...

The most relevant part was the change to luaB_load in 1.1.104 which simply disabled bytecode loading.

Never enable in your Lua apps:

a. Bytecode

b. Debug, Io, Os libraries

...

Also, be very careful with sandboxing. It can be deceptively easy to break out of.

Everyone eventually learns this lesson, see: Roblox (removed bytecode almost a decade ago after, if I recall correctly, an exploit exfiltrated their server tokens), Company of Heroes (bytecode bug leading to RCE)

Luau should come with safe defaults, from what I've been told.

Yup. I'm the one who did the Roblox bytecode exploit that lead to it being disabled: the specific attack there was that getmetatable internally leaves the metatable value on the Lua value stack even if it ends up returning the __metatable locked message, which you could retrieve with a crafted bytecode chunk. I leveraged that into getting the metatable for the global environment, which was the entire Lua standard library, and crucially the same table across Roblox's script permission levels. Roblox used "context levels" for seperating priviledge Lua scripts, which interacted with the server API endpoints, and normal game logic. By poisoning the priviledge context's metatatable with my own functions I was able to capture the server endpoint URLs and accesskey, along with things like send arbitrary HTTP requests to their CDN from the server to steal any place file. This easily could have been arbitrary RCE on their servers instead: the crafted bytecode chunk attack for getting a r/w primitive from for loops was published a little bit before this attack happened, IIRC, but there was a lot of less CTF style writeups about Lua internals and I was, like, 15 and an idiot.

In practice it's really hard to sandbox arbitrary user controlled scripts. Even after this I found a half dozen other bugs in their Lua<->C++ bindings that you could leverage into server code execution. V8 and other browser engines still having JIT bugs and DOM manipulation exploits every other week should terrify any developer who thinks "oh I'll let my users do a bit of scripting".

> Also, be very careful with sandboxing. It can be deceptively easy to break out of.

Back in 2018 when Surviving Mars came out with mod support (no sandbox of any kind), someone asked about os.execute on Reddit I think?

I whipped up a mod showing a couple other fun things you could do. I must've panicked someone because they pushed a sandbox update out pretty quickly, then sent me a polite pm asking me to let them quietly know about issues before hand.

About a week later I sent the dev an email with 5-6 different ways to get access to _G

I’m confused about one thing.

It doesn’t follow to me, that since all clients are running their own simulation, Lua scripts must run on every client too.

If a client runs a Lua script, why can’t we just run it on their machine and propagate any game state changes (if the script adds an inserted, for example,) as if the player made those changes themselves?

If clients don't run the same code they will desync the moment their state diverges. I haven't played multiplayer factorio but I expect you can't even join a server unless you're running the same factorio version and mods as other players.

> If a client runs a Lua script, why can’t we just run it on their machine and propagate any game state changes (if the script adds an inserted, for example,) as if the player made those changes themselves?

The game already has to run Lua scripts as part of the simulation, potentially as part of in-game events which aren't directly triggered by players. A player running a script from the console is handled by that same interpreter -- making it run in a completely different operating mode where any changes to game state are replicated would be much more complicated and prone to error.

Or, from the other direction: the game's multiplayer model is all based around a replicated simulation where player inputs are fed into the simulation. Treating a player running a script as a special kind of event involving the text of that script is the simplest and most obviously correct way to implement that.

The mechanism Factorio uses is to sync user inputs, not game state changes (the reason isn’t explained, but I strongly suspect it’s because user inputs are less data; small inputs can cause big game state changes, but not vice versa).

If the user types a command, in order to preserve synchronization, the game must:

- Run the command on all other clients.

- OR it could sync changes made to the game for just commands; in other words, the other clients apply the changes caused by the command instead of running the command directly. But that would be an unreasonable amount of extra work just for a small feature and to make exploits harder.

- OR the server simply disallows clients from running Lua commands, which is the case for some servers.

I don’t get the second part though: why a map can store arbitrary Lua code that runs when the map loads.

Scenarios that don't require a mod ?

(Some time ago, Factorio did not have a built-in mod synchronization system for multiplayer, with the result that the most popular servers did not run any mods, but rather used complicated scenarios instead.)

Running the scripts outside the simulation and syncing their commands alongside user input would definitely work on a technical level.

But I think you're massively underestimating how much these scripts can do. Many mods would flood the network connections. And there would also be an awkward delay for all script actions.

> why can’t we just run it on their machine and propagate any game state changes (if the script adds an inserter, for example,)

Because that's an unbounded amount of traffic. You can reliably write data into RAM at many gigabits per second, whereas network connections are variable and many of them won't carry more than a few kilobits at the 99th percentile (note that you roll that 100-sided die like 30 times per second, so "1% situation" lag spikes are something you'd run into constantly)

I sometimes use Lua commands in single player to clear biters from a certain region for example, which removes many entities. Propagating those sorts of changes on multiplayer (or take a more plausible example: wave defense that eventually spawns loads of entities at once) would cause a big lag spike if you have a few players that all need to receive this data, whereas simulating it locally on each machine is no problem

Factorio multiplayer bandwidth is like a dozen kilobytes per second from what I remember, and this post agrees https://forums.factorio.com/viewtopic.php?p=125328#p125328 (couldn't quickly find an exact number though it must surely be out there). If you make it O(n) for every lua-touched entity in the game, it would quickly balloon into the megabits constantly and many mods would just not be viable for multiplayer for most people

Is it impractical to employ firecracker vm like separation to isolate untrusted code, severely reducing the impact of any bugs?

Browsers split their various components across multiple processes to provide isolation. VMs would provide even more isolation.

That's what they do. They add additional layers on top of the sandbox like jailing. You can jail your sandboxes, but it's not so easy to make that a multi-platform solution for gaming. I think for games I would just stick to a fast interpreter and apply some generally appropriate measures to discourage timing attacks.

I literally just downloaded a factorial demo to my work laptop. Is this something I need to be concerned about if I don’t play online?

No.

Thanks

Yes, but only because you might lose your job from playing too much factorio. :) the exploit was not a risk for vanilla unmodded single players, and has been patched in any event.

Awesome! Yeah we’ll see I may not be able to start playing until I get moved to a boring / less intensive project. I started playing and it felt identical to my day job which is why I wanted to try it but after a day of coding I just wasn’t up for it haha

I’m tempting the devil here, but unless you have a PCB layout job, (in which case this will always feel like work), once you get going it can be very soothing to watch your factories churn stuff out and ship it around on your rail network. The very start is a tiny bit clicky and takes a lot of manual labor. I often start with the nanobots mod for that reason. I’m sorry in advance.

bytecode is also architecture dependent

At this point, I have serious doubts whether bytecode and JIT systems, whether it be Lua in Factorio or JavaScript in Chrome, can ever be verifiably secure. I think we would all be better off if, like Apple’s Lockdown mode, we can disable anything JIT on a high stakes system.

I don’t blame Factorio though - this (anonymous?) researcher is 100x developer material.

How is JIT relevant here? Unless I missed something, the attack uses straight-up malicious byte code, it doesn't exploit the JIT compiler.

JIT and Bytecode are two sides of the same coin, in my head. JIT also uses bytecode in some languages like Java.

It's much worse than that because of the complexity around JITs, behavior of hardware and speculative execution. Proper sandboxing is really hard, and I suspect that if people really want security they would disable JIT in general. Even simple ones like pcre2. Personally I have disabled Firefox's JIT (I believe it's called ion in the settings, but correct me if I misremember) for a few years now. I've never had any trouble with any websites so far. It's not instant loading, but it's close enough.

... but I don't know if I would lump bytecode with JIT. Bytecodes don't need or use RWX execute segments, for example. Lots of your favorite JITs do, for speed.

In this case someone generated malicious bytecode that the JIT compiler would not generate.

I would argue JIT is dangerous because it requires dynamic memory without the NX bit set, so if you manage to smash the stack (find an exploit) you can execute arbitrary code easily (leverage the exploit). That's a different dangerous than running malicious bytecode.

JITs can still function with an NX bit; You just have to halt execution to modify it.

This doesn't generally apply but Apple has a bunch of hardening in place that means you don't just have an RWX mapping hanging around:

https://developer.apple.com/documentation/browserenginekit/p...

On top of that they're working on (I haven't checked in a bit) having the JIT compiler be entirely Out-Of-Process. I don't think a lot has been written about that publicly but there's a few breadcrumbs like https://developer.apple.com/documentation/kernel/oop_jit_con...

All modern compilers use bytecode, in one form or the other.

I would take a look at BPF.

The issue here isn’t things being “verifiably secure”. languages like js and lua run in a sandboxes environment where the only functions and operations that are permitted are those explicitly added by the host environment. Those sources languages are easily validated as correct code.

[edit: I realize I should clarify something “correct” and “verifiable” here do not mean “bug free”, it means ‘cannot interfere with or violate language or environment state, memory, or other invariants’]

The issue here is that the hosting environment is allowing the user/attacker to provide the bytecode that is generated from the provably correct code. That byte code is not itself verifiable statically, and is not verified at runtime (and it might not even be possible to).

This is not to say that bytecode is not verifiable - Java, .NET, or even WASM (which is intentionally very low level) are verifiable bytecode environments. The issue is that a byte code must be _designed_ to be verifiable (and early Java bytecode was not due to JSR or similar iirc). Lua’s bytecode is designed for execution, and so allowing arbitrary bytecode execution is not too dissimilar from a JS engine allowing a website to provide direct access to their bytecode interpreter which would be similarly catastrophic.

The issue is not really about the verifiability of the bytecode but about the interpreter checking the invariants. JVM bytecode is intentionally designed to be verifiable so that the inner loop of the interpreter does not need to care about whether the operation is executed with correctly typed operands (which in the JVM case would be highly impractical and would essentially mean that you need twice the amount of memory). You can design a system where the bytecode is not verifiable, but instead the checks are done at runtime, CPython works that way.

Yeah absolutely. This exploit is akin to V8 allowing websites to supply its own bytecode (not WASM -- talking about the internal bytecode here) to it instead of Javascript.

JIT has not been secure since Spectre and Meltdown.

Not sure why this comment got downvoted or flagged all the way to ghost text. Seems reasonable to me even if some people disagree.

(I actually disagree also but I can still respect this take, which was clearly also well intentioned and spoken.)

Reply on news.ycombinator.com