1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond (2001)
gamedeveloper.com
gamedeveloper.com
I basically learned to code thanks to a few helpful and extremely patient IRC members in the Spring RTS engine[1] community (some of whom I spot in these comment sections on occasion - hi!), and deterministic lockstep simulation for online play was one of the first ideas that totally blew my mind when I learned about it ("b-but that means all the "random" numbers on every computer need to be the same! and so do all the floating point calculations across different hardware!"). Of course, to me at the time, that was deep magic that I thought I'd never be able to fully understand.
[1] see https://springrts.com/ - or check out http://zero-k.info/ and https://www.beyondallreason.info/ for games on the engine that are still pretty active
It made for incredibly funny replay of matches if you happened to own a newer version of the map, as the game starts to diverge from reality and enter some parallel universe where players do nonsense actions and movements.
For NBA Inside Drive 2000, I developed all the ball physics to be deterministic and "reasonably" accurate (25 years ago, in a world without physics engines). This meant that after a player made a shot and the ball left their hands, the trajectory, scoring or bouncing off the rim, was purely controlled by deterministic physics. This in turn meant that we could foresee what was going to happen, and feed the player AI with that while the ball was in flight.
Obvious use was to move characters with high rebounds stats towards the place where the ball would bounce after not scoring, so they're more likely to pick the rebound.
[edit: for shooting the ball, the instant the ball would leave the player's hands I computed the perfect trajectory to hit a shot, and then based on the player stats, I'd nudge the trajectory off center. It was then purely up to the physics to determine if the ball went in or what]
However, you can't predict the other players' actions, and those actions can insert an arbitrary (and unpredictable) number of random() calls between now and when your next action executes, so practically there is no gameplay advantage you can get from that avenue (AFAIK).
Though maybe a cheat engine could simulate a few branches of possible inputs a few steps ahead and adjust inputs when it discovers split-second advantages?
But there are bigger exploits. IIRC, the client sends separate actions for buying a unit/tech/building and paying resources, and the other clients don't enforce that they go together. This allows malicious clients to buy without paying. It's been seen a few times even in the ranked mode of the Definitive Edition remake. https://www.reddit.com/r/aoe2/comments/fjjudp/cheaters_are_a...
This way of doing things made sense for the time, and the execution was impressive. It was pretty reliable, nobody wanted to cheat anyway, and yes lag was a big problem but there wasn't another good way to do it back then. Nowadays they should have something else IMO.
Finally found a genuine use for the blockchain, since the server is only used for the lobby!
Mine virtual AoE gold and record it in the distributed ledger. Primitive path finding is also AI, so now you just need some investors!
The only exploits that are inherently applicable in the case of a simultaneous deterministic simulation model are those of the information-revealing class (e.g, "maphack"). This is where a standard AH system comes in.
The obvious one is that the client knows where everything is on the map, even when hidden by fog-of-war - so client-side hacks can reveal that.
A hacked client can easily desync the game too, or stall it and cause it to time out if the player is losing.
LHC@Home had big issues[1] with AMD and Intel machines giving very different results to the same work unit. They traced it down to the exp() function behaving different, and ended up using a library[2] for anything more fancy than basic arithmetic.
[1] https://lhcathome.web.cern.ch/sites/default/files/OPENshort....
The id of the node acts as the deterministic seed. Really handy when building views that slice and dice data, it feels like you’re interacting with a database.
> At first take it might seem that getting two pieces of identical code to run the same should be fairly easy and straightforward
I found that a little funny. I would certainly not assume that. Synchronization problems are one of the scariest areas of programming in my opinion.
The biggest downside is mentioned in the article: If one person lags, everyone lags. In an 8-player game, the chance that at least one person lags was pretty high. I used to play on the Mac version of GameRanger, a small community. That was 2008, when reliable internet wasn't as common. Players had personal reputations, and those who lagged too much were often excluded from games.
You'd generally want an authoritative server to accomplish this. Having it also execute the inputs prevents most cheating, too.
I've built and licensed a time traveling variant of this model (GGPO style) and spent years with thousands of concurrent users playing my games. And they operated just as I describe.
Company of Heroes is more of a traditional AoE style and will happily queue up unit orders for a minute or more while it attempts to reconnect or catch up.
Yes it relies on the players trusting each other a little to avoid extreme lag, but there are plenty of ways to ruin this kind of game besides lagging. It's not like CoD where a few useless teammates won't make a big difference, while it didn't have automated ranked play like Csgo. (but the recent DE remake does, and yes it needs better netcode)
Time-traveling netcode... I know Slippi does this, and it works great.
There just aren't many modern multiplayer games where slow players can negatively impact the experience of others, and that's because the industry has learned how terrible the experience is.
And that's for players playing in good faith on bad connections - abuse by inducing latency was a big thing in the past. Think a player that runs out the clock when they're a move away from being checkmated.
Basically the only time you'd ever want to operate the system to pause if a player fell behind is during a tournament, where fairness is more important than player experience and they know that going in. And even that, IMO, is questionable.
Consider for instance the situation where-in one player's simulation enters the spiral-of-death, under the policy that you're proposing this player is effectively removed from the game; I would say this is a substantially worse trade-off -- I know that I would personally prefer some mild annoyance of brief pausing as opposed to a player leaving the game (and in session-based games, as RTS's generally are, a team-mate leaving often ruins the entire game)
Abuse is indeed a problem that most games I've seen of this nature don't address, and while there's no perfect solution, it can be quite effectively curtailed. In the systems I've designed, the basic principle is that each participating player has an allocated amount of wait time which serves as a quota that is charged while-ever the game can progress/run, but cannot, due to them being in a wait-state (connecting/loading/reconnecting/stalling/explicit pausing/whatever). There's some extra logic for handling the oscillation of wait-state.
While a client is catching up, you run the sim in a loop and don't render the game.
The deterministic simulation code is only a small part of the overall game logic, and you can run it much faster than real time.
If a client is slow enough to genuinely experience a spiral of death, the game will not be playable for them, and "mild annoyance of brief pausing" is not remotely what the other players will experience if they're forced to stay in sync.
Wait time is the common solution in older games to prevent abuse, yes. But it doesn't prevent somebody with high ping making a game run at half speed.
Go play Company of Heroes 2 with a packet loss simulator. Even as the bad client, it's totally playable. For everyone else, it's silky smooth.
I don't think you'll come back and tell me you prefer the implementation in AoE II Definitive Edition.
In the end this is still a decision regarding how to handle player's falling behind; too far and they're effectively taken out of the game as they're no longer interacting within a sufficient approximation of the current state.
Under what design is "high ping" going to cause the game to run at lower "speed"? The only possibility I can see is where-in the turn period is, for some bizarre reason, also the granularity of the simulation step and you're also adapting the turn based on player latency (in aid of fairness, usually)
Having implemented such a system myself, a few times now, it has in practice performed sufficiently well.
DE is garbage in general so it's not really a fair comparison.
If you could simply wait a few seconds for them to catch up, they'd also be able to catch up without a global pause.
In any case, we're going in circles. I came to this thread to claim that "If one person lags, everyone lags." is not intrinsic to this kind of net code, and you disagreed. Company of Heroes and my own games are existence proof of my claim. I'm not really interested in arguing about it any more - millions of players can't be wrong.
That makes it very different from this model in my eyes. You wouldn't have every client running the game in sync. The AoE devs had this dilemma and chose the other path.
It wouldn't surprise me if the AoE remakes do this to work out who won the game.
Definitive and HD Edition both seem to use nearly the same netcode as the original game. And the whole thing lags if one player lags. Yes it handles situations where all players leave, so there's a server doing something, but the glitches in Elo changes hint to some weird edge cases. I even found an easily triggered bug in HD Edition where both players would gain Elo at the end of a ranked game.
The netcode was written so that players only send their inputs along with a timestamp (The "timestamp" actually being a frame counter). The server would then relay those inputs with the timestamp to other players. The client maintains snapshots of the game state and would then re-run the game logic from that timestamp given the new input.
I handled clients falling behind by making the server reject any inputs that were more than 500ms (Really, 30 frames) behind the current time. The server would send a message to the client that essentially said "You're lagging, here's the current frame counter and current state of the game"
Granted, this approach really only worked for my game because the entire game state could be squished to ~300 bytes. But it had the added benefit that it prevented players from cheating using a lag switch. They couldn't have a malicious client that sat and listened to what other players were doing and then send a bunch of back-dated inputs to put itself in an advantageous position.
But you could keep it strictly as cheat prevention, which keeps it very close to the AoE model.
I wouldn't start a game in 2023 trusting any input from players, including win reporting - but I understand if their hands are tied by a legacy system, and the players vote on outcomes.
Yes, the remakes should've had new netcode. They had full freedom to redo it, no legacy compatibility to deal with. Maybe they wanted to totally emulate the old experience, but ya know, I can just play the old game. DE at least seems to be working out the kinks better; HD was a total mess even after years of patching.
As you describe, during the game the clients are enforcing the rules, so a player can't eg teleport their units.
"Sure, blame it on your ISP!" https://www.youtube.com/watch?v=HGBOeLdm-1s
If your ping in the lobby was 1000ms (effectively "timeout") or you took too long to download a map, most players would just kick you preemptively
"Bro I'm just playing on my other iMac G3!" kick
I think that internet multiplayer is possibly one of the hardest things in game dev (use an engine that gives it to you!!!) despite feeling like some little sideshow.
I imagine Blizzard and Westwood have similar stories from the 90s.
there's this for Warcraft 1 development
I don't understand this. Running the whole simulation locally on both ends means that a modified client would have access to the whole game state, and I don't really see how you could patch that out.
Anyone have any idea what they actually did? Try to detect modified clients? Obfuscate the game state to make it harder to interpret?
I worked at MacSoft during the Age2 and Age3 days. The ports were faithful to the windows versions, but cross platform hadn’t been solved at the time because of this problem.
It also made long game play sessions longer because the calculated state kept getting more complicated.
There was one particular Mac OS X update that broke math interoperability for multiplayer because they changed how math worked on the OS. This meant we had to bundle a common math library to ensure the game the game states would line up, preventing unnecessary CRC checks.
Which would let you know what your opponents are up to, without impacting state.
Also, I used to play the Mac version a lot. It was great! Shame they made the remakes Windows-only.
[1]: https://redrocket.club/posts/age_of_empires/ [2]: https://www.youtube.com/watch?v=3tl8AjDfnBk
They run brilliantly through Steam on Linux, fwiw.
From maphacks to automatic reaction and microing units
The model would crash the game (and world editor, that's why we have to spawn it during runtime) when displayed, but it wouldn't get displayed when under fog of war, so you'd put it in a place that is impossible to be seen by a player under normal circumstances. But if someone uses a fog of war cheat or a maphack, it'd crash for them.
Of course it won't prevent you from more advanced hacks which e.g. modify the client and display an overlay of the enemy units rather than just revealing the fog of war.
Another option would be for the peers to only send data that the other should know (fog of war), but that's a lot trickier. Because you then need to figure out how to validate the unseen data wasn't cheating, too. You might be able to do something with this today, because storage, computation, and bandwdith has grown so much.
Maybe store all the local state changes, and when a block becomes visible, send its current state first, and then stream the history as time permits; the other side would accept the state initialy, until it could fully validate it. You would need some way to keep the randomizers in sync and fair, too.
But that only protects from seeing what should be invisible; you could still have computer enhanced movement and maybe enhanced display of data attributes that weren't supposed to be human visible.
However for something like an RTS, the amount of data in game state updates can be prohibitively large to transmit to clients. The deterministic lockstep networking model described in the article is a solution to that problem. In that system, the only data transmitted is input, and each client updates locally, so it does require them to have a copy of the entire game state.
I take it as a sign that I'm in a good healthy place mentally when the most stressful and anxiety inducing thing my brain can conjure up is me being rushed by 3 units in the dark age.
So as much as this is a fascinating piece of history and an impressive technical solution to the constraints of the time, I think modern games ought to move past it.
But why? As far as I know mega-hits like Warcraft III and it's new, updated, version, "Warcraft III: Reforged" which came out 20 years later still use that technique.
The benefits do go well beyond being able to "send" hundreds of units across the wire: a deterministic game engine allows to create tiny replay files and, very importantly, allows to find and smash bugs way quicker.
Having the next game state being a deterministic function of the current game state + player inputs is great.
What would RTS games win by "moving past" that? To do what instead? How would you then implement the replay functionality? You'd also invariably run into a class of bugs which would be hard to reproduce but which would be trivial to reproduce using a deterministic engine.
From a latency point of view you're not gaining anything either: you need to receive the other player's units position anyway. So what's the difference between receiving the hundreds of unit's position or receiving the player input that created these unit's position? Just compute them, deterministically, as soon as you get the player's input.
I think it's also worth sometimes revisiting the assumptions made for the current algorithms and approaches in use. The original design was for dial-up modems. The networking landscape is completely different now. Maybe some of the original assumptions are no longer valid and a different set of tradeoffs is worthwhile.
The bottleneck is player input which is the most overestimated bandwidth stat in gaming. It's mouse movements and a couple of keys strokes per second. Top Starcraft players are in the 300 actions per minute range, that's still just 5 per second.
You can encode that very compactly. 2 bytes for each unit ID, source and destination. So 1000 units would be just around 4K, if they all start shooting at the same time.
After that, you can rely on that in most RTS games how units shoot is deterministic.
The subject already came up here on HN and some posted about games using deterministic engines before AoE.
The reason I did it is I had a bug which happened ultra-rarely and couldn't figure it out so I thought a long time about this and realized a could make the game deterministic and that would maybe allow me to record the bug happening and then be able to replay it. I found that by myself: back in 1991 I had never heard of anyone writing a deterministic engine back then. So I took a few days and rewrote the engine to be fully deterministic. Sure enough it eventually caught the bug: some case where the hero would clear a level after having fired two shots at once, which was an extra (by default it only had one shot at any time). The shot still on the previous level would continue to "live", invisible, in the following level, and would corrupt the memory. Classic.
Oh the memories to see that article again!
> Game Developer was first founded in 1997 as Gamasutra and has strived since its inception to be a leading resource and reference for game development and industry knowledge. Following the shift from Gamasutra to Game Developer in August 2021, the site has maintained that mission while embracing the in-depth content its namesake Game Developer magazine is known for.
(Same for the NIPS conference. But oddly not for the science journal PNAS.)
One day sex positive triple-breasted feminist whores of Eroticon Six are all the rage the next you're sunsetting harmless Sanskrit puns. Y'all strange.
For example, being able to micro manage archers to dodge other archers, and time their shots is something that wasn't possible previously.
RTS entices people into a strategy game. But the games are so similar that everyone but the top 1% of players will do better by simply following a predefined strategy and executing it as tightly as they possible can. And that sucks. Nudging guys a few inches to the side is the opposite of what makes the game fun for most people.
I'd like to see RTS games that randomize game parameters each time, requiring you to actually strategize.
I'd argue that most people that kept playing aoe2 online and kept it "alive" all care about that.
> I'd like to see RTS games that randomize game parameters each time, requiring you to actually strategize.
aoe2 already has randomised maps, I think that's why you actually get less strategy (in a sense).
In games with fixed maps, you see much more interesting one-off strategies (people like to call it cheese[0]) because you can plan everything down to the second. You'd plan & practice strategies on specific maps in specific matchups against the "standard" build orders.
AoE2 has these types of strategies too, but since the maps are always slightly different, you can't plan your building placement etc ahead of time.
There's also MegaRandom [1] which takes the randomisation to the next level. But I'd argue micro matters significantly more in that map, since you can end up in "unfair" situations where you need to outplay with micro to stand a chance.
> Nudging guys a few inches to the side is the opposite of what makes the game fun for most people.
That's how these games stay popular for 20 years. It's like complaining about strafe jumping in quake, it's what kept the game alive all this time.
I'm more of a fan of the in-game, on the spot strategizing that randomized maps force you to do, than the out-of-game meta-strategizing of fixed maps, where you do comparatively limited in-game strategizing.
I didn't say they didn't?
> aoe2 already has randomised maps, I think that's why you actually get less strategy (in a sense). In games with fixed maps, you see much more interesting one-off strategies (people like to call it cheese[0]) because you can plan everything down to the second. You'd plan & practice strategies on specific maps in specific matchups against the "standard" build orders. AoE2 has these types of strategies too, but since the maps are always slightly different, you can't plan your building placement etc ahead of time. There's also MegaRandom [1] which takes the randomisation to the next level. But I'd argue micro matters significantly more in that map, since you can end up in "unfair" situations where you need to outplay with micro to stand a chance.
The random maps are random within a very narrow set of parameters. They don't change the strategy. Practicing standard build orders is what 99% of players will need to do. I agree, at the very top level there is a level of strategy again. But in that grind to the top, the way you do better is by simply narrowing the time to get your first rush online. There's very little chance to actually change your gameplan.
> That's how these games stay popular for 20 years. It's like complaining about strafe jumping in quake, it's what kept the game alive all this time.
These games have not stayed popular. RTS has not done very well as a genre. You've got Starcraft and AoE2. And that's kind of it. AoM, AoE3, the shittier spinoffs, and AoE4 haven't made particularly big splashes. Nor have other franchises.
The thing is, these games are really fun to play casually, or in single player. But playing competitively where people are going to do stuff that works is so different and not compatible with what people liked when they were introduced to the series.
Consider a comparison to fighting games, where it's also a hard requirement to be obsessively deep with the game to make any progress. Fighting games have still seen a ton of success with many popular new titles.
edit: I was super deep on the AoM scene and mildly deep on the AoE2 scene; dropping off well past the title's prime but sometime before AoE2:DE
edit edit: I can also recall that the majority of online players back in the day would like the setting but seem to lose interest in playing normal maps and eventually settle on silly scenarios. It's not surprising that MOBA games have really taken off as that's what a lot of these scenarios were in some form; but worse.
https://i.imgur.com/xbbXKdq.png
Additional you can add mods such as 'Random Costs' where units and buildings have random costs so standard build orders are impossible. Because the game is still being updated, people are making new maps and mods, there are often new strategies being discovered frequently.
But to me, the best part of the game is playing with a group that are all a similar skill level. Fortunately the game has a match making system letting my mates and I get paired up with others.
The data we have suggests that people with your opinion -- "people want RTSes just about the strategy, not about control" -- are dead wrong. The most enduringly popular RTSes are mechanically demanding games: StarCraft 2, Age of Empires 2, StarCraft 1.
If you remove the control aspects of an RTS, you get something that's closer to turn-based strategy. Nothing wrong with turn-based, but that's just not what most people come to RTS for.
I think this data suggests the opposite. These titles are all more than a decade old. They have endured, but the genre has died. People often find it really compelling to get into these games and do the singleplayer stuff. The multiplayer experience didn't live up to that. Limited strategic choices and a demanding micro are not what most people liked when they first entered the genre.
There are some indie titles that seem to understand this.
What you're missing is that there have been many titles that have reduced the mechanical demands in pursuit of "greater accessibility", and have been even less popular. That's been most RTSes over the last 15 years, really. They're always talking about how they wanna get rid of the clickiness or base building or what have you. "We're not gonna be APM heavy like Starcraft!" they crow, as they quickly fall into obscurity while Starcraft remains.
It's not wrong to want the genre to be more accessible or fun, or less frustrating, but getting rid of technical and tactical options by reducing control of your units is generally a bad way to go about that.
> People often find it really compelling to get into these games and do the singleplayer stuff. The multiplayer experience didn't live up to that. Limited strategic choices and a demanding micro are not what most people liked when they first entered the genre.
Nah, what points the way forward is SC2's co-op mode. That was extremely popular among more casual players -- being more popular than ranked ladder at least for a time -- despite getting only limited support as a new feature in LotV.
And the way SC2 co-op worked is that it had the same mechanical inputs of course, but it actually had less strategy, not more; similar to campaign AI's, the AI in co-op mode is largely scripted and predictable, no scouting and very little reacting required. People loved it!
I'm not convinced this is accurate. I'm of the opinion that by the time AoE 3 came out the fire was gone from the genre because people did not like playing the multiplayer.
Excessive micro sucks, but that's not really the main thesis of my argument. It's more so that the traditional "Random Map Battle" mode is really devoid of strategy for the majority of players because you have so few real choices to make. You don't get to do much strategy. You probably have a build order for several minutes. And then want to rush. That's almost always the optimal move until you're very, very talented. I can't think of any competitive multiplayer focused RTS games that tried to address that successfully.
> Nah, what points the way forward is SC2's co-op mode. That was extremely popular among more casual players -- being more popular than ranked ladder at least for a time -- despite getting only limited support as a new feature in LotV.
> And the way SC2 co-op worked is that it had the same mechanical inputs of course, but it actually had less strategy, not more; similar to campaign AI's, the AI in co-op mode is largely scripted and predictable, no scouting and very little reacting required. People loved it!
I'm amused because I feel that was MY point. Single player (and naturally also co-op) content against scripted scenarios was the best part of RTS games. It's where the genre was born. You get to ask yourself "What am I going to do" and come up with a plan. My point is that you DON'T get to do that with Random Map Battle RTS games in competitive multiplayer. Instead you largely just get punished for not doing the meta plan; or worse, get punished because your execution of the meta plan is worse.
Well, it is. They've been a lot less successful. "The fire was gone from the genre" isn't how things work. SC2 co-op is a good example of this -- if something is good and polished, people will play it, have fun, spread it around, etc.
In contrast, you could look at the CoH series, which is one of the many games that went for being less APM heavy (albeit there's still a fair amount of micro involved). CoH2 is decently popular, sure, but much less so than AoE2 or SC2. The CoH/DoW branch is one subgenre of RTS that's less demanding, but you could look at the C&C and TA-like branches the same way. And yeah, they're moderately popular, but less so than the more demanding games. Why?
> It's more so that the traditional "Random Map Battle" mode is really devoid of strategy for the majority of players because you have so few real choices to make. You don't get to do much strategy. You probably have a build order for several minutes. And then want to rush. That's almost always the optimal move until you're very, very talented. I can't think of any competitive multiplayer focused RTS games that tried to address that successfully.
This is just an argument from ignorance. The truth is that even APM-heavy games like AoE2 or Starcraft have plenty of strategy. Well, maybe not the really cheesy games where someone immediately goes all-in, but the ones that go beyond a few minutes, yes.
Like, these words
> You probably have a build order for several minutes. And then want to rush.
don't even make any sense. In the case of Starcraft, if the game is several minutes in, attacking is no longer a "rush". It's just...an attack. Which you'd expect people to do in an RTS. And even early attacks aren't necessarily rushes.
The words you've put in here is like stumbling into a discussion about Counterstrike and claiming that the game is nothing but people using the AWP forever and ever, or that you can win every game of PUBG or Fortnite by doing nothing but hiding. You're simply wrong, and advertising that you don't really know what you're talking about.
> I'm amused because I feel that was MY point. Single player (and naturally also co-op) content against scripted scenarios was the best part of RTS games. It's where the genre was born.
If that's what you like most, that's fine, but it's true that these involve less strategy, rather than more. You don't have to outsmart someone, you don't have to scout them out and react, all the while they're scouting you out and reacting to you. The computer's on rails and there's usually a dozen different ways to exploit how dumb the AI is. That can be good fun, just like any other single player game, but it's not very strategic at all, because your opponent isn't actually strategizing against you.
> My point is that you DON'T get to do that with Random Map Battle RTS games in competitive multiplayer. Instead you largely just get punished for not doing the meta plan; or worse, get punished because your execution of the meta plan is worse.
People get 'punished' if they have poor strategies or poor execution of strategies, which is as it should be, yes. There's nothing special about being "on meta", what's meta is usually just what people have found to be effective -- at lower levels, the professional meta tends to be markedly less dominant, with all kinds of other strategies being effective that wouldn't be at higher levels. I'm considerably higher ranked than average, but I still lose to plenty of stuff that would never work at a pro level.
It's relatively easy to spot when a game has a problem with low skill ceiling: the very best players have difficulty distinguishing themselves from other top players. So if the top player plays the 100th best player, in a game with low skill ceiling the top player might have a 55% chance of winning or so.
This is very much not the case in AoE4. And if AoE2 would add auto tracking arrows tomorrow, you wouldn't see the skill ceiling drop appreciably. You'd see players with strong micro but weaker macro and strategy fall in the rankings, and slower players with better macro and strategy rise.
It’s amazing how times change, now I get annoyed when my ping is higher than 30ms.
300-350 was more the norm for where i was.
That's because 2sec of ping is used to request 124 fonts, each using one char.
Sitting on my couch using Wi-Fi over cable, I have 18ms to Google.com – call it a full order of magnitude better than a disk up user despite Wi-Fi adding 7ms.
Also, some early broadband networks backhauled too much traffic to central locations. I recall commonly seeing traceroutes from my home in orange county, ca going to kansas, and then coming back to servers in los angeles. I still see some traffic that leaves my metro area only to come back, but it's less frequent, and it's usually only going one or two states instead of halfway across the country.
- The AI system is not part of the deterministic simulation. This was surprising to me, and after contacting one of the original programmers it was explained that it was due to a desynchronization bug that the "AI and network programmers weren't able to fix it in time".
A consequence of this design regression is that, due to the AI now being authoritatively run by the designated host player, network congestion issues arose which lead to a clear series of progressively more aggressive optimizations to reduce egress traffic. This primarily consisted of a very simple filter (mentioned in the article) which dropped duplicate commands in the common submission path, meaning it applied to both local user and AI commands, along with batching of AI-submitted commands which would be flushed at rather arbitrary times.
I'll note that I've restored AI being deterministic in my project.
- A rather obscure determinism bug resulted from their compiler's implementation of a few CRT routines, namely fsin/fcos and a few others, which leveraged the specialized ISA instructions of the same name. The problem being that these transcendental functions are beyond the scope of the IEEE 754 spec. and ergo are hardware implementation-dependent. In practice, contemporary Intel/AMD chip families produce bitwise the same result, however those around the time of AoE are known to diverge on results to some small margin (as confirmed by an Intel engineer on a thread I came across while researching).
- The game employs a dirty-update system for rendering, not only that it's at a scanline granularity. This was something I was very pleased to see, as it's a exceedingly rare to see such an important optimization in games of this era (although common in earlier eras)
- There's some "interesting" naming conventions, one being prefixing member variables names with "value" -- there's even a "valueValue". Very little consistency in general in this regard, a reflection of independence between teams working on different components.
- While there are attempts at validation of input into the simulation (albeit woefully inadequate), it relies on ad hoc inclusion of PID (player ID) fields within commands. This is entirely useless, as this information is not authoritative and controlled by the players, permitting them to "spoof" the contextual information required for validation.
This is one of the more perplexing aspects of the engine, especially given the necessary information about the origin of a command is ofcourse available.
(As this information was later publicly published by other individuals I don't see a problem with elaborating on it here as I have)
- An example of missing the wood for the trees: session information goes through an ad hoc compression for its wire form (just bitpacks fields) to conserve bandwidth, however the architectural choice is to synchronize this session state by just having the host broadcast the state --dirty or not-- every 200ms, flooding the network pointlessly (atleast in the 18.8k days)
- While Age of Empires is somewhat notorious for its poor multiplayer performance (notably when contrasted with say, the recent "Definitive Edition"), and while its implementation of this synchronization model is certainly rather juvenile, it should be understood that the final MP gameplay issues are primarily due to the choice of a peer-to-peer topology over the public internet, which at the time was the most reasonable.
While superficially P2P may seem like it should achieve the lowest latencies for instance, the reality is that the primary determinant is the characteristics --jitter, delays and packet-loss, reordering, ect-- of the path between two hosts and a P2P architecture means there's n*n paths (network egress/ingress paths are typically asymmetric). In contrast, a server/client model you not only have far fewer routes, but datacenters are located at critical points in the network, roughly analogous to comparing travelling A->B along a freeway versus via the maze of residential streets.
- The state checksum "algorithms" involved are bordering on useless, atleast for state-tracing (such as when debugging desync. bugs.). They appear to have been devised by way of believing doing "a bunch of random bitwise ops" constitutes sufficient mixing -- a quick test demonstrated that csum collisions were not just possible, but occured sometimes for over 80% of inputs.
--
There's a lot more that could be said but I feel that's enough for now.
The thing is that this is perfectly reasonable if your network infrastructure is known. If you have fixed bandwidth and deterministic packet sizes, then you can do math and know what the behavior will be. Determinism is good! Also this assumes the network is single purpose. Which it was! For games in the 90s it was! This isn’t bad design, it was good design for the network infrastructure people would have had at the time!
I've been told the source code for it leaked some long time ago and has floated around for the past two decades or so. While most people contend that EA had lost the source code to the Command & Conquer series games many years ago when the studio was closed down and assets were shipped to "DICE" in Stockholm
However, I have been entertaining the idea of supporting other deterministic (RTS-) games, adapting them the same way --rearchitecting the multiplayer system/code-- as I have done for Age of Empires II, providing a unified platform for these games. The first candidate that came to mind was the Red Alert series.
- On the highest difficulty ("Hardest"), AIs are bestowed 500 to all resources on next-age research completion. - Work rates for researches are skewed to be slower for the lowest-difficulty setting.
Are you the OpenAge author, or maybe Voobly? I've never read such detailed info on AoE's internals online. Thanks for the post.
> In practice, contemporary Intel/AMD chip families produce bitwise the same result, however those around the time of AoE are known to diverge on results to some small margin
That little bit of validation I needed for why I avoid AMD CPUs. (not entirely serious)
The project in question --a platform for AoE2 multiplayer /w general game improvements-- is currently under development, although the core game/MP has been functional for over a year now and our little beta-testing group plays regularly.
Actually suffixing! i.e. objectsValue.
He put out a comment effectively saying "I forgot not everyone has a T1 line. I'm going to order Dial-Up for real-world testing"
https://zoo.cs.yale.edu/classes/cs538/readings/papers/terran...
Maybe the link can be replaced? Also can (2001) be added to the title?
>>sarcasm overload
Install uBlock Origin yo
(1) https://apps.apple.com/us/app/adguard-adblock-privacy/id1047...
Android has things too. Though I’m not experienced enough there to remember what to recommend.