Building a Distributed Turn-Based Game System in Elixir
fly.io
fly.io
I couldn't really ask for anything better, as solo dev in less than 100h basic distribution solved(one process per map, one per player. Maps are spread around the erlang cluster using horde, if a map crashes players seeemlesly reconnect in less than a second, and the map may actually be on a whole different system, especially when the node becomes non viable) Basic monster ai, fighting logic. Respawn/monster drops, extensive ddbuggability using observer (gui tool included in erlang, allows me to send messages to processes/see their status/ what they do, maybe kill them in the whole erlang cluster ) Can just use livebooks to do more extensive programming (just connect to cluster, execute a mark down files code blocks, see results live ingame and in the livebook)
It's a very great and easy language for this. I may consider moving couple of things to rust using rustler tho for the fun of learning about nifs
Any chance you mind sharing that code on GitHub?
https://gitlab.com/zen_core/zen_core And u need assets from the original game to use it
I made a small example clip to show how I use livebook to simulate player load(they use exactly the same genserver as real players, just that the packets are already deserialzed )
https://cdn.discordapp.com/attachments/819516082154242048/83...
This is worst case for it, the other maps do not see any load by this and its basically a single core bottleneck, also it's very unoptimized code 500 processes sending messages to a single map process
Or if they spread across all 70+ maps we see a good utilization of every scheduled but still most map processing spending most their time idling
5 times the players(250), tops out at ~35% scheduler utilzation
An example:
(FYI there is no actual game here, just pick emojis, place them in the arena and watch them fight to the death via a poorly designed combat system. Yes, those placing emojis other than you are other players.)
This was a prototype that I built to see if real-time multiplayer games were feasible. Apparently LiveView manages high tickrates decently enough to be a valid solution for (quasi) real-time games. This game for example runs at a whopping 8 ticks/s! It can handle more, as I've tried developing with faster tickrates, I think as high as 24 t/s, but I want to avoid server strain. With enough HTML/CSS/SVG wizardry you can get away with quite a lot. But the most amazing thing was the fact that there was no need to fiddle around with state syncing. Everything just works! I have nothing but praise for it.
Try LiveView if you have the chance!
In the "Let's walk it through" section, it seems like the Player-to-LiveView connection will process user input (e.g. a Tic-Tac-Toe move) and update the UI to acknowledge this, at which point the user can be assured that the LiveView server accepted their input. But it seems like this happens before the GameServer has also accepted the input. What if Player 2 made a conflicting play and their change was accepted by the GameServer before Player 1's change reached the GameServer?
Given, in Tic-Tac-Toe, the game is simple enough that this is neatly avoided: each regional LiveView server has enough information to only allow the current player to make a play. But in more complex applications, how might you (anyone; curious for discussion) handle this?
One answer is something like: The LiveView server is effectively producing optimistic updates, and the GameServer would need to produce an authoritative ordering of events and tell the various LiveServers which of the optimistic updates lost a race and should be backed out.
Not sure I understand the question but, I don't see how this would happen.
On the BEAM it's processes all the way down. There's a process for that instance of the game, which is basically a big state machine, and 2 processes representing the client state, one for each player.
When the game (process) starts, it expects a message from player 1 (process), then one from player 2, and so on.
If there's a client timeout or network disconnection, the player process affected crashes, and if the app has been architected well, the other player process and game process are in a supervision tree, so they crash as well, perhaps notifying the other player that the game has ended because of a disconnection from the other peer.
But none of this will accept a move from the player 2 when it's player 1's turn.
This is very interesting - I'm pretty unfamiliar with BEAM. Does this "processes all the way down" span across machines/VMs?
From the article, it seemed like there could be two players, each connecting to different LiveServer instances (on different VMs/hardware in different geographic regions) which in turn communicate async via one central GameServer.
In the article, it seems like a message from Player 1 to LiveServer 1 doesn't need to wait for the message to also reach the central GameServer and be acknowledged before LiveServer 1 acks the change back to Player 1. This seems to allow races, since the central GameServer is the source of truth but the Player1/LiveServer1 communication can complete a message/ack round-trip without waiting for acknowledgement from the GameServer.
I guess an alternative would be for the system to require a message from Player 1 to be passed to LiveServer 1, then passed on to the central Game Server which acks back to LiveServer 1, which finally can ack back to Player 1 -- this means that Player 1 would still need to pay full round-trip latency to LS1 and then to the GameServer for any action.
Thanks for any light you can shed on this!
> The browser click triggers an event in the player's LiveView. There is a bi-directional websocket connection from the browser to LiveView.
> The LiveView process sends a message to the game server for the player's move.
> The GameServer uses Phoenix.PubSub to publish the updated state of game ABCD.
> The player's LiveView is subscribed to notifications for any updates to game ABCD. The LiveView receives the new game state. This automatically triggers LiveView to re-render the game immediately pushing the UI changes out to the player's browser.
So you can see that when Player 1 does an action, the action is sent to the GameServer. Player 1's UI is only updated when the GameServer has published the new game state via PubSub back to Player 1's LiveView process, that pushes it onto the client. So there is the latency of going from client to LV to GameServer and back again, but there is no race possibility.
yes for example if u had a process named on a different Machine(Node called in Erlang) called "Alice", u could from a different Node send it a message using the Node Identifier as additional parameter example:
[coolest_node | _rest_of_nodes] = Node.list()
Process.send({Alice, coolest_node }, :hi)
https://moosecode.nl/blog/how_deltacrdt_can_help_write_distr...
Have you played games like HOMAM 1 or 2? You can't do anything when the CPU is playing its players. You can watch where he goes and what he does but that's it. When he is finished then you go.
When there is a network error - some message or please wait... or loading spinner message should be shown in the meantime.
For turn based RPG games or Chess etc. this is a non issue.
Of course, real time action games etc. is not a good idea for this technology.
It's all done on the server.
I mean, 100 ms between a click and a cross appearing on screen is not great user experience, but it's not even the worst. If you're writing a game, a little client side prediction is a good idea.
But if you have a form with instant validation, or any old regular UI, that is not necessary at all. The only built in optimistic UI functionality on Live View is disabling a button when you press it and wait for the server to respond, to avoid double submissions.
Arguably because you're trusting the client and essentially the built or built-in behavior is therefore optimistic by default. Then hopefully validating on submission server-side.
Though maybe I’m wrong and there has been some new developments to address this, I wasn’t following too closely.
There are also use cases which are a bad fit for LiveView: Animations - animations, menus, and general UI events that do not need the server in the first place are a bad fit for LiveView. Those can be achieved without LiveView in multiple ways, such as with CSS and CSS transitions, using LiveView hooks, or even integrating with UI toolkits designed for this purpose, such as Bootstrap, Alpine.JS, and similar.
https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#m...
When you need to go to the server, you go to the server. There's no other way around it.
I would be curious to hear how you solve this in other stacks? SPAs, whatever, when they need something from the server, they reach for the server.
in liveview server renders me the list view, i click details, server renders me details view, i click delete button, server renders me the list view.
if there's big latency/connection error/etc between clicking delete and getting back the rendered list - user just has to wait.
in spa i could optimistically assume that delete worked and render the list that i already have cached without the deleted item, allowing user to continue working immediately and if there was a disconnect/error - i could retry that delete in the background without bothering user, only prompting them after some number of retries.
don't see how could i implement this workflow in liveview.
pushEvent, pushEventTo (from client to server) [0]
push_event (from server to client) [1]
[0] https://hexdocs.pm/phoenix_live_view/js-interop.html#client-...
[1] https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#p...
so, "just use JS"?
> In case of any errors, notify the user, refresh the state .etc
so, "just use JS"?
every time you say "just use JS" you're diminishing the usecase of liveview because if i need so much js logic - why do i need to also use liveview if i can just use a framework/environment where i can share codebase between client and server seamlessly and have full control.
In case of any errors, notify the user, refresh the state .etc => This would all be done server-side and the client side would simply react automatically. The client side code in this case would have been minimal.
I don't think I'm diminishing anything. For quite a few years I was neck deep in React/Vue world. Now that I'm actively using LiveView, I can properly compare the differences between both approaches, cons and pros. For any new project, in the majority of cases I would pick Elixir with Phoenix LiveView instead of Elixir/Phoenix (backend) with React/Vue (frontend).
> For any new project, in the majority of cases I would pick Elixir with Phoenix LiveView instead of Elixir/Phoenix (backend) with React/Vue (frontend).
This could just be recency bias. New tech is always exciting, old tech is always linked to memories of all the issues you’ve had in the past.
> This could just be recency bias. New tech is always exciting, old tech is always linked to memories of all the issues you’ve had in the past.
I still maintain some React/Vue apps and work with on a daily basis, so it's not a distant memory.
I like choosing the right tool for the right job. For example, I would still choose ReactNative over 2 different code bases for a mobile app for a small team that needs to move fast. For the 5% of cases when that wouldn't do you would need to go native. I see the situation similarly with LiveView. It's hard to beat its productivity & power in 95% of use cases.
Its nice tech, but once you start introducing JS again to improve UX you really start asking yourself why you didn't just build it with react in the first place.
If you really want to do it, you can add 3 lines on your project and that will work with any CRUD page you're building, I don't think that's unreasonable or difficult to do.
Edit:
Actually thinking about it, if you just made a form for that delete button and a phx-disable-with="" on the row it would probably work straight away without any JS hook.
If you want a delete button per row, no code is needed and the phx-disable-with will work out of the box, if you want a global delete button on the top which deletes multiple rows front-end first before acknowledgement (with checkboxes + delete like in Gmail), 5 lines of JS maximum in a hook and you're set.
that you can't even acknowledge that there is a gap in functionality between liveview, a fairly opinionated framework for server side rendering and fully fledged client-side frameworks tells me this is not going to be a productive conversation, so i'm out, bye
I've worked for years with React and Angular and I don't really miss anything with a LiveView-based stack. LiveView features gives you 90% of what you want out of the box and for the rest it's fine having a bit of JS here and there to ensure a good experience.
Do you mean when websites just need a full refresh because they lost their requests on some callback and no-one implemented recovery across the 5 levels of callbacks or something more specific?
Having latency on such lowlevel interactions might make the UI feel sluggish as a whole.
Most examples are to show cool stuff you can do, they're not production vetted. Like most JS examples out there don't really mean that people should be publishing live credentials with their bundles.
I imagine in most cases one would leave everything that is not behind a logged in status as normal routes/pages (signin, landing, contacts, etc). Or if not that those would be things requiring a socket/real-time interface anyway.
For the interactions, I don't think you even need to use alpine.js. Plain setup on DOMContentLoad, CSS Dropdowns/collapsibles that are replaced on JS load, proxying LiveView DOM/Morphdom events (if needed) so other components (even vue,react, etc) can listen to them, and CSS animations.
import { setup_live } from "./phx/setup_live.js";
import { setup_dropdowns } from "./interactivity/dropdowns.js";
import { setup_collapsibles } from "./interactivity/collapsibles.js";
import { setup_links } from "./interactivity/links.js";
import { setup_inputs } from "./interactivity/inputs.js";
function setup() {
setup_live();
setup_links();
setup_dropdowns();
setup_collapsibles();
setup_inputs();
}
document.addEventListener("DOMContentLoaded", setup);
Then LiveView is mostly for your admin dashboards and logged in users views, where it makes it pretty easy to do real-time feedback type of interactions/views and spa like navigation. Since you have proper auth and identification on the user, you can just log them off, rate-limit, block an account, and close their socket if needed.
- Is it a vendor lib ?
- It's not but is some particular file that is big enough to not make sense including on the root layout?
- It's neither, but functionality that can trigger multiple times and should be only once and only on those pages because it can conflict? or some variation of that?
I think they're all solvable but what makes sense will depend on those, but also on how you're using LiveView (is it LiveView only for logged-in users/some auth, can you set those on the live_view layout...)
But in some cases this is a problem also in spa's, where you have to use a snippet to check if the lib has been loaded, if not add a script tag to the body, or load it through js, etc...
[0] https://hexdocs.pm/phoenix_live_view/js-interop.html#simulat...
[1] https://hexdocs.pm/phoenix_live_view/js-interop.html#loading...
But yes as soon as you're on the internet you'll often feel the delay if your app is interactive.
The problem is that it's a bit random, because the network and the VM performances are never totally linear.
I remember implementing a countdown (using 1s send_after()) that would work fine most of the time, but sometimes there would be some hiccup and the countdown would stall just a bit and then process the counter in an accelerated fashion, which was terrible from a UI point of view, so in the end I did it in JS except for the update once the end reached.
In a nutshell, you compete with other players on creating freestyle piano solos over backing tracks using a MIDI keyboard. Players and audience members then listen through the solos and vote for their favorite.
The Elixir/Phoenix stack has been an absolute superpower for building my game backend. Some examples:
- Phoenix channels as a wonderful abstraction over
managing WS connections. Implementing basic chat took ~3
minutes on the backend.
- The ability to model game logic with a finite state
machine in a GenServer. The lifecycle of a game round is
progressed forward by receiving incoming client events.
- The ability to have many game servers
running simultaneously as a dynamic pool under a
DynamicSupervisor. Games can end and new games can be
created, all isolated and under watchdog-like
supervision.
- ETS as an out-of-the-box memory cache for session data.
Persisting user data between pages and sessions without
needing to deal with an actual persistence layer.
Game (desktop chromium only): http://midimatches.com/
Gameplay Video: https://www.youtube.com/watch?v=UVD2wOCB_jE&t=41s
The way it worked was fairly straightforward: a user would click a link from their account area and it would include their Stripe customer ID in the request (all our users had already had accounts created due to our main booking system). The app then span up a new GenServer to represent that customer, which would pull down all the data it needed from Stripe's API as soon as it initialised. There was also some generic data stored in a "global" GenServer to cover stuff like plans etc (I can't remember how I had that refreshing, probably via webhooks).
Then as the user went through the subscription process or management process, any changes they made would be made through calls to their own personal GenServer, which on write would first write to the Stripe API, then update its own cache with the new state returned from Stripe to ensure consistency. These GenServers were kept alive by a timer set on interactions with it, and would automatically shut down 30 minutes after the user stopped interacting with it, discarding all the data it held. Then when they next return, it fetches the state from Stripe again. It also listened for webhooks and would update running GenServer instances with data it received to ensure they were consistent, and just ignored any for users that weren't currently running.
Overall I was really happy with it - it was really performant due to the data caching but also didn't suffer from staleness issues :)
I haven't used it yet and am curious about new offerings.
That said, the guide linked from this post requires more config than the competing options.
For example, Render also handles clustered deployment with libcluster setup, and it's very, very simple to set up. http://render.com
Liveview is crazy powerful if you can keep the latency down.
The less latency the better, though, of course. I'm seeing roughly 80ms, which is very good.
HTTP/2 has this by default, and allows multiplexing multiple requests on a single connection, so even making multiple parallel calls won't require establishing new connections.
Basically, performance between AJAX and web sockets should be comparable in most cases.
Fly apps get anycast IPs, so it's basically bgp getting you to our "nearest" edge proxy, then we connect you to the closest vm from there.
https://debug.fly.dev will show you which region you hit first (the fly-region request header).
--edit--
I snooped on your HN profile and found Australia. We don't have an instance of this app up in AU, but we could. You're likely connecting to us in Sydney, and LAX is most likely the fastest over our backhaul from there.
Bonus points if you could bring the node down again when usage drops.
kurt fly.io
My first choice was node but the more I use Javascript the more I hate it.
Also keep in mind OTP (the Erlang stdlib) is sort of one company's kitchen sink helper lib. It's got some gems, but do read other people's opinions about which are worth using. I wish it's docs included more about when not no use it.
Edit:
Definitely look up "contexts". They're the Phoenix word for module that handle storing/retrieving a piece of data.
If you're writing an app with low perf requirements consider using the db directly for this with no in memory persistence. I did this with a low usage internal dashboard and it's was dead simple.
Edit2: Thanks to reply, previously said contexts were an Elixir concept
Gen_statem is erlang standard, for state machines
https://hexdocs.pm/gen_state_machine/GenStateMachine.html is a a fine wrapper for that
If u have finite states I would go for a state machine
Context is the practice of creating a public API for your (database) models, opposite to having your controllers access directly your DB and database objects. Gives you better testability, better isolation, better code architecture.
So instead of (in pseudocode):
def change_password(user_id, new_password):
user = User.db.get(id=user_id, is_active=True)
user.set_password(hash(new_password))
user.save()
send_password_changed_email(user)
render(password_changed.html, user)
def change_password(user_id, new_password):
# The Accounts module hides all the complexity and implementation details
user = Accounts.get_user_by_id(user_id)
Accounts.change_user_password(user, new_password)
render(password_changed.html, user)
state,events, transitions as the parent said, are very well buildable with state machines, since thats basically what they explicitly are.
and since we are talking about games, it would be more logical to use than, contexts, loading/saving state to a db is just secondary nature of a game, its more about the state transitions and events of player input and game logic.
LiveView is the nail in that coffin. It doesn’t help that Ember, though I STILL love the technology and framework, is deader than dead community-wise except for a few silos. It’s very tough to find talent. And the benefits just aren’t there anymore compared to using LiveView. I still use ember on about 8 projects but no more.
Have others with this stack reached similar conclusions?
This was well-timed for me - nice to have a reference working config to play around with, especially for clustering. How do you find the BEAM handles a distributed cluster across multiple regions? Are there any oddities that crop up due to inter-site latency?
Animations are still better off handled client side, though you can go far with web components and liveview updated HTML: https://github.com/hmans/three-elements
Definitely not it's intended use case, but it works perfectly on regular Internet latencies in the 100-300ms ranges.
Whereas interactive elements just require an update in 100ms or 200ms after the user pressed a button, which is not that hard to achieve.
For server rendered animations, all the frames might be delayed due to geographical distance, but the time gap between the arrival of individual frames is not affected by geographical distance. The server is not waiting for the client to request the next frame, it’s always pushing.
For the user, it’s similar to watching a streaming video after letting it buffer a bit.
For interactive elements, every interaction has to go through the round trip. So the gap between interactions is affected by distance.
Server rendered games would suffer much more than server rendered animation.
For example, if you want a 60 fps animation created and the server is far far away, like the other side of the globe that could be an issue or if your connection is bad.
With liveview, you cannot create offline apps without javascript for example. That is a big limitation. But since most apps don't really require offline capabilities that is not really a big issue.
I have however tried to use a liveview hosted in Sweden from Japan (which is basically on the other side of the globe) and I didn't experience any noticable latency for simple html updates.
I think if you're making a game, you'll still need to use javascript. But if you are just making a simple crud app you may not need it for at least most of the stuff.
I still think creating modals or things that the user expects to be instant should perhaps still be done in javascript but for all the server requested data liveview is probably faster than your average SPA.
Can anyone ELI5?