Game Programming Patterns: Event Queue (2014)
gameprogrammingpatterns.com
gameprogrammingpatterns.com
https://www.gamasutra.com/view/feature/131503/1500_archers_o...
You had high-ping 28.8k internet, and you wanted to be able to have immersive battles in a world with hundreds of units controlled by several players. There's no way you could send real-time information about the status of every object in the game. So they didn't. They made the game simulation completely deterministic and then simply shipped user input around on the network. User input was processed on a slight (couple hundred milliseconds) delay (to allow for network transmission speeds) and executed simultaneously in "game time" on everybody's machine, so the simulation stayed in perfect sync. Voila, no worrying about lag.
Here's an extensive writeup about the concept:
Thanks to the link above, I found the exact phrase: neural antedating/backdating. Ah, it's nice to have resolved that question.
They clearly have some phenomenal tech chops in the company. Iwata hand rolled a compression algorithm and Nintendo were in the internet game waaaaay early and had aspirations of being an ISP with their Famicom Modem in the 80s.
That said their web infra just seems so janky and backwards - I've heard excuses like the friction wrt online gaming keeps it family friendly and helps disincentive trolling.
Nintendo has also always been more focused on in person social which works well in Japan where console market penetration is very high, gaming in public is normal (both socially and public crime/mugging doesn't exist) and population density is high. Monhun (Monster Hunter culture) being a great example of JP cultural gaming trends that didn't export nearly so well outside of Japan. (did ok-ish in Asia). [--aside: Must have been so ace to have monhun parties after work /sigh--]
If Nintendo could make their estore more optimized for UX (think Amazon or Apple App Store / Google Play Store) I feel like they would be a money printing machine.
Of Sony, MS, Nintendo, Nintendo are the only pure games company and those marketplaces are fantastic cash cows...
I remember when the Wii U Smash was coming under criticism for the unbearable lag, the developers released a statement in which they literally said: we tested the online play in development, and it works great. Are you fucking serious? How can they care so little?
If I recall correctly, there was another benefit. You could save a multiplayer game by just having all peers dump the current game state to disk and resume it later. Since a big SupCom game could take HOURS this was a great feature.
However, there were 2 issues with SupCom's implementation.
1. It wasn't perfect and sometime hours deep into a big game, you would get a "desync" error and it was unrecoverable.
2. Everyone needed to be able to run the sim at the same speed. So the game could only progress at the speed of the slowest player's computer. And SupCom was VERY compute intensive.
> So the game could only progress at the speed of the slowest player's computer. And SupCom was VERY compute intensive.
It might surprise you to learn that these statements are still true! SupCom:FA (still playable by buying a copy and through the third party FAF launcher; they have even fixed most of the desyncs, it's very stable these days) is probably still the largest scale RTS out there (even planetary annihilation only feels larger). And it still requires a top-tier gaming rig to play.
The reason for this is pretty straightforward, CPUs actually haven't gotten faster for well programmed, cache aware, tight-looped CPU bound operations in the last 15 years. And in fact there are arguments for why something like SupCom:FA would be difficult to make today given the industry's reliance on game engines. And games are still rarely threaded more than SupCom was (e.g. game logic thread, render thread), so until someone can get back to the low level excellence of SupCom:FA and then design and implement a concurrent game logic loop for it that actually parallelizes well, I think it will remain the king of high scale RTS for quite a while.
https://youtu.be/DEhp5eCwgSI?t=3304 (to see why this game is awesome)
It always made me a bit sad that the comparatively low-scale Starcraft seemed to suck up so much of the space in the genre.
Your comment brings back fond memories of playing huge supcom games at a local LAN cafe. Those were some of the best times I've had gaming.
With that said, most games do have a maximum tick rate, so in a very misleading way you could say they are turn-based ;)
The observer gets the replay log (cheap), and broadcasts the pixels out (expensive) to the world with very little impact on the game play for the active participants.
Low bandwidth broadcast isn't really relevant I think to esport success -- the "complete" replay aspect to even 1 viewer (the caster) however might be
You both say that the game state is sent to a non-player machine that creates the actual video stream seen by the viewers.
Much more notable is the impact it has on casting, where it's presence is clearly felt -- casters are granted much more freedom, and can be much more thorough and descriptive, when complete replays are available (versus simple dumb video).
We agree about the technogoly -- we disagree on its impact
There are times gameplay needs don't allow that model to be perfect, but it is the norm.
The main reason for the deterministic lockstep architecture is the cost of sending game state -- in a game with 10 player characters and some particles, it's sufficiently cheap. In a game with 10,000 player characters, not so much.
Determinism is a bitch to maintain, so afaik, no one tries to do so unless they must.
https://github.com/LMAX-Exchange/disruptor
https://github.com/LMAX-Exchange/disruptor/wiki/Performance-...
I've started using a variant of this in my .NET Core projects and have found the performance to be astonishing.
Meanwhile, Doom Eternal has no main or render thread at all, and instead uses a massively parallel jobs system. https://www.dsogaming.com/news/doom-eternal-does-not-have-a-...
And some of those small scale hobby/indie devs may later end up working for "the big leagues" as well, so the extra context can benefit them too.
You can argue about when it is appropriate to use threads at all. But, if I'm going to use threads, I use a threaded queue for communication exclusively.
The big games will also prepare data for rendering in parallel (eg culling and sorting and whatnot, although much if this is also done on the GPU).
(Going by GDC talks, the rendering teardown articles and just what I see online from Unity/Unreal. I don’t work in games myself)
That's a big issue. There is a surprising amount of back and forth between objects in a single step of gameplay/AI.
And, generally gameplay code tends to be a big mess of wild and ever-changing requirements from gameplay designers, extreme time crunch and short term (1 game then burn it) goals. Ivory tower software architecture it is not...
Clustering physics into "islands" is common practice though.
> https://max0x7ba.github.io/atomic_queue/html/benchmarks.html
I am also interested in the productivity and safety afforded by high-level languages in this arena. Dealing with memory and threading at the same time is not something I like to do at a low-level.
I found the abstraction this was originally developed for - processing of financial transactions with latency as the primary constraint - as an excellent analogue for UI event processing. Latency is also a huge concern when the user's eyeballs are in the loop.
Just imagine all the things that it would be possible to make automatic if all the events that happens inside a business environment where in a queue.
And don't even think about how it would be great if the same queue were open to outside providers.
My previous company was doing trading and made the outgoing orders available on the ESB. It's was quite handy since all teams could subscribe to the trade topic and do their own processing: trade checks, accounting, risk, etc.
It's a bit old school though, I mainly see old corporations with that architecture. The thing is, it ages quite badly most the time.
Fast forward 5 years, and every single team is publishing stuff on the ESB "cause it's super easy to publish and consume data from the ESB". Before you know it, you have a full team dedicated to scaling and maintaining the monster.
May be I'm bad at phrasing this, but the point is, sometimes you have perfomance problems because you could write better code, or use better tech, or tweak requirements a little bit, or do something else a little smarted. But sometimes you have perfomance problems simply because you just do a lot of actual work.
Once you have it, it's very tempting to use it for everything. But of course the more you use it, the more features it needs to have.
Let me illustrate that with a example compiling some of the patterns I have come to see at companies that overly used their ESBs:
Say Bob is working at a trading company and sets up an ESB where he publishes the daily trades of the company.
Soon, the trading desk learns about that, and decide that it's actually cool to use this for trading, they just have to listen on the ESB, and send the trades accordingly. That's the error pattern 1: making the ESB business critical.
The next day, the risk team learns about that ESB thing, and decide its very handy to perform post-trading checks by just listening to the trades flowing on the ESB. So they setup a system that listens for trades on the ESB, check that the trade is compliant with some limits, and send an other message on the ESB to let everyone know that this trade is validated. This is error pattern 2: cascading messages triggered from other messages.
The week after that, the security team learns about the ESB, and decides its very insecure to let anyone see the trades of the company, so they start implementing access control on the ESB. This is error pattern 3: now you have an overly complex layer on top of the ESB to decide who can see what and who can publish what.
Rince and repeat patterns 1, 2,and 3 for 5 years and here is the situation you end up with:
- The ESB is not the easy and handy system it was in the beginning. Since it has become the de-facto standard for publishing information in the whole company, it has to support features for _all_ the company use cases. There is access control to publish per topic, access control to listen per topic, multiple bindings of varying quality for each technological stack/language that each team in the company is using. The company of course is not capable / prepared to maintain a software of this scope, so the ESB is crippled with bugs that nobody can fix, because, you know, the infrastructure team cannot fix their groovy scripts using the ESB cause the guy that wrote them left. And the marketing team has some interns using the excel plug-in but they don't have time to rewrite them this year. The ESB is now partly un maintained, because the company relied totally on it without having the capacity / willingness / foresight of understanding how intricate it can be to update something that everyone use.
- The ESB is now very slow, because it was so tempting to publish anything of various interest on it that everyone did it. The problem is that the ESB is also critical for the company, so the whole flow of message is now slowly moving and overflowing, requiring endless tuning and tentatives at scaling it better. Of course 80% of the messages on the ESB are actually not listened to by anyone, but since nobody really knows who listens to the published messages, it's very tempting to just _not_ stop programs from publishing, ever, because god knows if some random team at the other end of the company might have a program reading these messages.
- You most likely have now an IT team dedicated to maintain the ESB. They are squeezed and pressured by the business teams to keep the ESB fast and easy to stable without requiring them to recode all the crap they plugged on it. On the other end, the other IT teams are pressuring them to update the ESB to support <place your language/stack here>. Of course the ESB team has no incentive to make any improvement whatsoever to the ESB, because that would definitely crash most of the crap the less technical teams of the company plugged to it. But the ESB team is the de facto guardian of the temple of the ESB, so everyone ends up frustrated by the situation.
---
I'm not sure I did a good job at explaining the various problems here, but basically, the one size fits all that ESBs are promoting is often not a future proof choice.
The reality is that you don't want your whole company coupled to a single system like that. Otherwise your system will be as good as the worst user of it.
-but-
My experience there 100% confirms what you say about it not being something that scales well.
It was a relatively modestly sized firm by headcount, as trading firms go, and there was a corporate culture of absolutely intense inter-team and inter-departmental communication. One absolutely would not dream of subscribing to an event stream without first talking to the team that maintained the program that published it. Any changes to the event stream - both the grotty little details about what was being published and how, and the grotty little details about what was being consumed and how - would be preceded by a discussion among all the people who were working with it, to make sure that everyone involved continued to have a complete picture of the interactions involving it.
That level of communication, which I do believe was essential to the bus's long term success, just wouldn't scale to a large company. Nor could it have been maintained at a company that had a more relaxed attitude in general. Nor could it have been maintained at a company where programmers are allowed to believe that most of their time at the office can be spent with hands on a keyboard.
What would though? An ESB could be a way to enforce a standard and slow down gung-ho devs & teams using ad hoc solutions for every problem. But say an ESB is not the solution -- what is? As the company grows, overhead and friction become more important. I'm not sure "every piece connecting to every other piece in whatever way" would help with scale, rather than compounding the problem...
As far as specific things to try go, domain-driven design is my personal favorite off-the-shelf mental framework for dealing with these sorts of things. Especially the concept of bounded contexts. Embrace Conway's Law; recognize it's not a criticism, per se, it's also a scaling strategy.
I've never seen DDD successfully used in any company I've worked in, but that's probably a shortcoming of my own experience. (Likewise, I've never seen TDD or Agile or lots of things people often mention in their blogs successfully used. Again, this is probably my own problem!).
addendum: to be fair I've never seen a completely working ESB either. Always a plan to build or deploy one, never the finished thing ;)
I see them fail more often than not, too, and one thing that's consistent about every failed implementation I've witnessed first-hand is that non-developers who are involved in or influence product development weren't engaged with, bought into, or properly trained in the framework.
But if everybody it in the company it is using it, maybe it is worth to have.
Must be well though of from the beginning, like proper ACL and notification of reception (so we know what messages some other people listen to) but overall it does not seems so bad.
In your example, the trading desk would either need an intern to communicate the trades or some software developed ad-hoc. Similarly for the second examples. The data need to be moved from the trade to the risk analysis someway. Either custom software or some human need to do it.
For the third pattern, I agree that should be backed into the EBS, but you need ACL anyway. Either you let all the people in the company see the trade or you have ACL implemented somewhere.
I definitely can see the problems, but I can also see the benefits...
I'm not sure I understand why making the ESB business critical is an error. I mean, it's a key piece in the architecture of your system (say, like a database would be for an app that uses one). It makes sense that it's critical if it's central to your solution. It's also unsurprising if it requires a dedicated team to maintain and monitor, much like any piece of critical infrastructure would. Am I missing something?
What's the alternative? Multiple ad hoc, p2p, uncontrolled connections and streams between arbitrary components of your solution, of varying quality, and many that require maintenance of different kinds. This works for a smaller software with fewer connections, but how does the effort scale as the system grows?
But in the case of this discussion, we should not forget the "E" in ESB. These aim at being company-wide.
In the example I wrote above, we could have a bus for flowing the trades, the risk department could use a database to store their check results, the devops could use an ad-hoc time series database for their metrics, the interns with excel could just read plain files exported from other systems, etc.
The key here is that technological diversity inside a company is not a bad thing. Sure, it does not seem as "unified", but in the long run, it gives each team its own little realm of responsibility, room for standalone technological improvements and tech stack switches, etc.
I guess overall there is a fine equilibrium to find between having a totally uniform stack used by everyone, and having each team using drastically different tools. ESBs can make you fall into an extreme without realising it, because at first, it may look like a grandiose unification plan.
This totally nails it on the head. The problem with Entreprise service buses is the same problems that every major unification plan runs into. It usually executives looking for one ring to rule them all. This misses all the complexity that is going on in the business and deprives the teams that are actually solving those problems the independence to solve them the best way.
Agreed! I love working in companies where there is margin for teams to choose their own tech (within reason). It's just that it's hard to know where to draw the line. At one company I worked in, the infrastructure team had a terribly difficult time developing company-wide solutions within acceptable deadlines because the tech and standards where all over the place, the result of a policy of "everything goes, as long as it keeps the company running". This works until you reach a point where some rule must be enforced company-wide... and then chaos ensues. Rules can be anything company-wide: tests, privacy, backups, automated checks, disaster recovery, any kind of compliance, monitoring, etc.
Agreed about your overall point though, and that the ESB is an extreme.
The features, constraints, goals, etc. expected out of the ESB substrate by the different use cases becomes a nightmare of responsibility/blame shifting. Durability and replay? That's the ESB's problem. Load balancing? Yep, ESB. But it's screwing up the ordering guarantees! Well, make the ESB 'smarter' and we can just keep punting all of our problems to someone else. Etc.
For this to work, the ESB must be acknowledged as the critical piece of the architecture, and the team responsible must be empowered enough.
However my question remains: is the ESB the source of the problem here? If your company would world "like that" — as in, with all the circuimstances and limitations you mentioned — would using different systems really bring a better outcome than ESB?
Essentially you have a standard protocol and a set of message to exchange health-related information between the different softwares even if they from different vendors. This allows (for example) a doctor with software A to ask for a test on a patient, have a nurse see it on software B, have the sample sent to a laboratory which uses software C and then the result sent back to software A where the doctor can also check it. Meanwhile the signed document has been stored on software D.
That said FHIR especially is more than that and allows different approaches.
And yes, you're right, it is quite unruly (information being present in one field or another depending on the software and the like). In my experience luckily most of those issues could be handled at the broker level.
https://en.wikipedia.org/wiki/Enterprise_service_bus
Spoiler alert: it's not the beautiful, decoupled silver bullet it seems like. The concept makes a ton of sense, but good execution of it is difficult. The biggest issue I've run into using systems that use this pattern is throughput and latency. Most shops design and deploy their systems in such a way that as soon as there's a little bit of traffic beyond the norm, the whole thing gets clogged and backed up for hours.
[0] https://en.wikipedia.org/wiki/Linda_(coordination_language)
Think about an automated factory with tens of thousands of tools, each with hundreds or thousands of events, all from differing vendors. The only rational way to tie this stuff together is with some sort of event-driven messaging architecture.
https://lwn.net/Articles/608915/
https://fedmsg2.readthedocs.io/en/latest/
I remember having a play with it a couple of years ago, and it was nice to see the various messages flowing over it.
I'm not sure how HATEOAS is related though? This is about publishing and subscribing to events (or streams of data), not about API discovery or "resources" as documents. Or did you mean something else?
Main drawback is that each task would have to have a proper description, context and definition of done, otherwise it's bound to fail. Once you have those, any tool would do. But usually those three things are the hard part, not the tool/technology.
I’d like to add that similar queues are widely used in all multimedia, not just games.
This library https://github.com/Const-me/Vrmac/tree/master/VrmacVideo uses queues to coordinate 3 threads, one decoding video by pumping V4L2 kernel queues, another one decoding audio and playing it with ALSA, and the main thread rendering frames with GLES. Two of my queues are thin wrappers around mq_send / mq_receive Linux APIs.
I suspect the author knows more about this than me but plays a little fast and loose in their explanations. I feel it's important to have disipline in our vocabulary so that people learning the material can participate in more advanced discussions as they grow.
That said, I really appreciate the overall content and using the context of games for broadly applicable concepts is a great way to promote them (and funny as generally commercial games have some of the loosest "structure" of any code I've seen).
I agree with what you say. But I'd add the suggestion that this depends on the context to some extent. For example, in the world of hardware and electrical circuits, a "bus" and a "queue" have almost no connection at all. But in the world of enterprise software, an "enterprise service bus" is often thought of as basically "a bunch of queues and a message router". Of course that's imprecise as well, since it ignores orchestration... which just goes to show even more how fuzzy some of these terms are in practice.
I feel it's important to have disipline in our vocabulary so that people learning the material can participate in more advanced discussions as they grow.
I generally agree, but I think this may be a case where "that ship has sailed." :-(
https://news.ycombinator.com/item?id=23205511
This gets posted very often.
...it has some interesting content, to be fair, but I think there not much left to rehash again.
It hasn’t changed since 2014.
I'm pretty sure there are a lot of HN users today who weren't here in 2014.
--> https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Some kind of link from this gets posted every 2-3 months or so, since it came out. I appreciate you haven't seen it before... but.... this gets posted a lot here.
The previous (635 point) thread was literally 3 months ago (https://news.ycombinator.com/item?id=23203699).
There are a lot of articles being posted and so even if there were a feature "here is what you missed" I would not have used it :)
Does DES have some deep magical tricks or it is just plain commensense type of discipline?