Bootstrapping a multiplayer server with Elixir
elixir-lang.org
elixir-lang.org
Throwing together a prototype is instant, but the awesome part is how quickly you can turn that prototype into something useful. More concurrency? Done. Need to sock away some state somewhere you can easily find and deal with it? Easy. It's easy to write, easy to reason about, and gives you fantastic composable tools to put something great together. It doesn't feel like a codebase so much as an incredibly well managed and cohesive OS.
Most people might have experience with functional-ish features like HoFs and lambdas in imperative langages but Elixir, through Erlang, is much more of the package: immutable structures and bindings, expressions-oriented, patterns, limited support for iteration. It’s not pure or anything, but it’s closer to OCaml than it is C# or Javascript.
“I’ve used Array#map” doesn’t mean you know functional programming.
You might also be interested in the discussion happening on the associated Twitter thread: https://twitter.com/elixirlang/status/1420782381825503240
In my scalability testing, we could handle upwards of 1000 players in the same 50 nm by 50 nm square on my 8 core dev workstation. That's pretty far beyond the limit of what would actually be _fun_ to fly the sim with. It becomes pandemonium even with 100 planes in the same area. We could easily do a dozen pockets of 100 planes in the same area even on our current 8 core VM.
We could one day be smarter about this and only send you everything if your map is actually open, and otherwise limit it based on your field of view. That hasn't been a priority yet, though.
But it would make a great fit for an MMO. AFAIK there aren't any of those now levering Erlang / Elixir. If anything they're doing the opposite and trying to contain people in smaller and smaller instances.
How many players can you handle in close visible proximity?
What’s the bandwidth usage for spread out states?
What’s the bandwidth usage for the max player state in close proximity?
How much is client driven vs server?
Are you doing data compression on the plane states?
Probably easy to make predictions of a plane as it’s not changing speed or direction very fast, which where I see the low poll rate coming in.
Looks like it was a fun project to work on.
I want to ask: do you share the nearest-region data to all planes within a region only for informational purpose or is there a crash possibility and you take planes out of the simulation, etc.?
I mean like in other MMO games, can players hurt each other and that information actually affects other client apps? That is where, I feel (I am new to this too), the global state mutations become more expensive right?
1. https://thinkingelixir.com/podcast-episodes/035-x-planes-eli...
"this article explores why they chose Elixir and how a team of one developer - without prior language experience - learned the language and deployed a well-received multiplayer experience in 6 months."
I guess they have low constrains since they only need to support 10000 players at a time and they want a single instance for that.
- Erlang runtime is slow
- Memory consumption is high
- GC is not very good ( vs Java / C# / Go )
- high cost in term of performance because of immutability
It's not the kind of things you want for a game.
To give you some perspective on popular FPS games, they run between 30hz and 120hz with up to 64 players and usually only use a single CPU core. And they're doing gameplay simulation / state replication / physics sometimes AI etc ...
Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice? ( this concern is actually higher than the performance one )
Overall I think it's an impressive achievement to be able to do that for a single person in 6 month!
edit: I just read through the article, and it doesn't seem to say one way or the other. 99% sure they aren't running any kind of physics engine on the BEAM, but there isn't anything specific
Either way they had to re-implement Racknet which is annoying because from now on everytime they add / change something in the network layer they won't be able to re-use the client code in the server.
I'm curious on why they did not go with C++.
"Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++"
https://developer.x-plane.com/2021/01/have-you-heard-the-goo...
I did some experimenting with rolling my own networking layer in c++ and it was a lot of work, and tried to do it through unity but it seems to require quite a large investment in time to understand how networking fits in the overall picture of the game. But maybe I'm looking for magic where it does not or can not exist :)
I upvoted your comment to bring it out of 0 or less because these concerns are valid and critically make or break important when building multiplayer servers.
My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric.
> My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric.
They are handling 10k concurrent players with low resources usage.
You can build a “game” with a million “concurrent” clients that do absolutely nothing, to dealing with thousands where you have tick rates that make that impossible.[1]
[1]: https://www.planimeter.org/grid-sdk/api/Tick_rate_and_bandwi...
- Memory consumption in production peaks at like 300 MB for us
- Scalability (i.e., being able to go wide on, say, a 64-core CPU) was more important to us than raw, synchronous speed
- Garbage collection is not something I've ever had to think about with Elixir (which hasn't been my experience with Python and Java)
- Because it's a sim and not something like a FPS, we don't have to worry about cheating (what would cheating even mean?), so we don't have to do any physics on the server side
- Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++
This is such a rudimentary question in game development that I don’t know how you can’t answer it. Mainstream server authoritative design has been a thing since 1999, and probably even longer when you look beyond Quakeworld.
All it takes is a few malformed packets from a script kiddie and then you realize you have to do distance checking and other anti-cheat functionality.
“Cheating” in gamedev doesn’t specifically mean there is a win scenario and you need to make sure people don’t shortcut that and play unfairly.
It means in this case that someone can’t connect to your server and transport their plane to another nearby player and wave it around in their flight path to annoy others while they spam chat to grief people for fun and to produce a YouTube video.
And those checks are pretty simple.
Long-term we've talked about things like a reputation system, where flying responsibly would earn you points to get into a separate "world" filled with people who also want to do more serious flying.
Presumably X-Plane is small enough and "boring" enough that your only malicious interactions would be only lazy drive-by attempts, but those kinds of situations would be more concerning to me than someone modding the client to make their plane fly at Mach 7.
The criteria are also so brittle at those concurrent numbers that if the game decided to do any more than what they do today, you would face engineering challenges more difficult than other game server software.
As you deal with more and more concurrent clients there are calculable limits to how you can facilitate more features and how bandwidth and server frame time budget limits you.
There is probably very little the server is actually doing. Player state as seen from the server is probably just x,y,z coords, plane type, etc. Just the bare minimum to be able to draw another player in the game.
For that use case Elixir (or Go, or maybe even Python with more computational resources) is a very good fit.
A multiplayer server needs to run the simulation for every player and be the authority, the clients should simply collect input from the player and run a "lite" simulation to hide latency.
Of course, this leads to the standard RTS cheat — map hacking — because the client has to have everything by definition. But you have enough entities in an RTS that syncing full state, or even delta, gets ridiculous quickly, so it’s just something that has to be dealt with. Fighting games also tend to do something similar — deterministic lockstep with rollback — which also sensibly runs with no authority (again, you can’t teleport/wallhack because you’re only passing player inputs, and moving just your character is the same as the sim desyncing)
But my main point is that an authoritative server is not a guaranteed architecture for multiplayer games — the game design should definitely factor in (as it does in OP’s case).
Could you share a bit more about your experience in the industry? I think you and they have different use cases, which makes you reach different conclusions.
If you can split your workload in tons of actors the beam GC works great. If you need very large synchronous actors (even just a few) then it’s excruciating, so is splitting a simple single-threaded sequential workload into a concurrent mess just so the GC does not eat itself.
Of course you probably should not be using erlang/elixir then. But let’s not pretend the beam gc doesn’t have issues and pitfalls.
Something like simulation (e.g. physics), a sequential long-running CPU-bound process.
> Blocking on what exactly?
I never used the word "blocking", I'm not sure where you imagined it from.
> The point is surely that elixir/Erlang makes it difficult to write concurrent code with lots of global shared state in the first place no?
Never used the words "global" or "shared" either.
I used the words "sequential" and "single-threaded" though, which you apparently completely swung by. The entire comment was about non-naturally-concurrent workloads which would not trivially map to concurrent actors (or even not be parallelisable at all).
In which case the GC is just a fairly simplistic STW.
I sadly don't remember the name even though I've been scratching my head, but way back when (in the early aught when I first played with erlang, before SMT and Programming Erlang) one of the major "public" erlang codebase was a GUI-based tool which did basically everything in one enormous process, complete misuse / completely unsuitable to erlang.
It is a lot, lot worse yes.
Because of the purpose of Erlang / Elixir, and the way the runtime functions, the GC is rather simple (though not trivial): it's a stop-the-world generational (2) semispace collector. So on a GC run, the execution stops, the GC acquires a new empty heap, scans the stack (the "root-set"), traverses the tree of heap object, and each heap object it finds is copied to the new heap (the actual process is a bit different but that's the idea).
That works well, and the generational hypothesis applies nicely because Erlang only has immutable data structures so unlike Java and friends an Erlang object can not refer to something younger than it is. However it generates a lot of garbage as an "update" requires creating a new object.
It's quite simple, and has good (though not amazing) throughput, but it has horrible latency.
The trick is, the GC works per process. So the "world" it stops is a single process, and all the stack scanning and faffing about is per-process, meaning on a stop it might have to deal with kilobytes of data, megabytes at the absolute worst. It doesn't need to scan the entire runtime and go through gigabytes of heap as Java commonly does, because Erlang processes are shared-nothing (aside from the global ref-counted "shared heap" but that's a bit of a special case). This means with "normal" usages (normal for BEAM) a given GC run has very little memory to scan and not too much work to do per collection, it's essentially leveraging the other characteristics of the language to create an emergent concurrent low-latency collector, the concurrency and latency are not part of the design of the GC, but instead part of the system the GC is used in.
All of that falls over when you start moving away from lots of small actors, and towards few big actors. Then the size of the stack increases, the amount of garbage explodes, and the concurrency drops precipitously, because the GC's "world" covers a larger and larger amount of the program's surface.
IMHO it’s the opposite, at least in the context of games: In languages like Java and C#, the GC is global and freezes the world. The time would be in terms of milliseconds, while in Erlang the GC is per-process, in terms of microseconds.
So in terms of GC and latency, Erlang is not as good as C++ and Rust, but it should be better than Java and C#.
> Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice?
I haven’t dive into the problem but my intuitive tells me it could be done through NIF. It’s a pretty common practice to use Rust to compliment Elixir in performance hotspots.
Re: GC, the Erlang VM does it per-process ("process" in the EVM's sense, i.e. a preemptively-scheduled userspace/"green" thread rather than an OS process), so not only are GC operations fast, but they also only affect the process being GC'd. The immutability is also a factor here; since data is copied instead of modified in place, it's arguably a lot easier to know when to discard unused objects (namely: for an object to persist within a process, the recursive function(s) backing some process has to explicitly pass it along to the next invocation, so that's a natural place to discard everything else).
I think someone's trying to tell me something.
This is not accurate. Any function can print, can call a DB or a web service.
"Purity" (as in referentially transparent) is tangential to function programming. It can be achieved with other paradigms. That being said, functional programming does encourage it.
So all in all while stating "every function has no side effect" is incorrect, I still share your overall feeling.
https://github.com/X-Plane/elixir-raknet
It's not a full game server, but the "Usage" section of the README provides a sketch of what the rest of the server (the part that implements the business logic) looks like.
Games would generally be going against the grain if the langage, not easily leveraging its strengths and really needing areas where it’s weak.
A game server, however, could be good. At least if you don’t need very high performances (e.g. if you need fully accurate physics sim on the server then probably not, plus you’d want to run the game’s own sim anyway).