Lessons learned while developing Age of Empires 1 Definitive Edition | Hacker News Reader

Lessons learned while developing Age of Empires 1 Definitive Edition

richg42.blogspot.com

277 pointsdeafcalculus66 comments

Develop strong out of sync (OOS) detection tools early, and learn how to use them.

The way lockstep work is that the clients gives themselves a rendez-vous in the future and agree to compute one turn of the game, given the players' input of that turn, at that moment. It's very clever. The players input are sent to every clients and will be computed in the future in a deterministic way. So every client is computing the exact same game. Now if the clients doesnt compute the same game given the same players' input, the game is OOS and that basically should never happen because the game is dead.

To detect an OOS, a client needs to compute a hash of some relevant game data (ultimately "everything" ends up being represented by the position of the units (plus a dead state)), every turn and send them with its "end of turn" message to the server/clients. If clients disagree with the value of the hash, they are OOS.

The cool thing with lockstep is that you can record every turn inputs (which is fairly light) and then replay the whole game - because that's already how a game is played. So, when you have an OOS - or any bug, you can just replay the game until the OOS and try to fix it. Sometimes it doesnt work though and you may replay the game without detecting any OOS.

If you dont detect an OOS after a replay, you're basically in deep shit. I mean needle in a haystack of c++ kind of deep shit. Every client needs to dump every data of every function that ever ran in a log file, and then you need to reproduce the OOS and then compare the log files and then guess what caused the data to differ.

Of course if you want your game to update its state at a rate faster than the slowest ping time of the connected client then you've got problems. On LAN play you can get away with this, but over the Internet it's not as workable, especially if some players are on modem.

I remember one game that has crossplay between Linux and Windows brought down by a scheme like this, because it turns out that Linux and Windows default floating point libraries had slightly different rounding behavior, and eventually the pathfinding algorithm would zig on the Windows hosts and zag on the Linux hosts and desync the clients.

You can run your updates faster than the slowest ping. You just need more command latency. So when you, say, command a unit to move somewhere, there's an agreed-upon future time when that's going to happen. This time has to be longer than it takes for the command to get to everyone. If it's not, then you stall. If command latency is very high, the game feels very unresponsive though (and tricks of showing local/visual-only confirmation of your commands only goes so far).

You can't run updates any faster than the slowest machine can run them, though, or again you stall. This was a bigger problem on Age 1 and 2 where render rate and update rate were linked. On Age of Mythology and beyond, the render rate was decoupled from the update rate, so updates could happen more slowly while animations and so forth could still look smooth via interpolation.

I used command latency to do multiplayer for an RTS\Tower Defense on mobile (https://youtu.be/tRAdI0x7L7c?t=300 - think Battle Cats but in 3D). All units were AI controlled, and placing a unit meant it had to go through a spawn animation. It meant the game worked with ping times in the range of seconds. Add in some positional interpolation and events for when each unit changed targets and it worked great.

As an aside, fixing that problem should be feasible. Linux (or rather libgcc or whatever the case may be) uses an open source floating point implementation (and not virial like the GPL): https://github.com/gcc-mirror/gcc/tree/master/libgcc/soft-fp

With enough finageling you could utilize the libgcc floating point implementation on Windows and make it consistent everywhere. Or you could take the wiser approach of avoiding floats outside of graphics code and using fixed point instead.

Does that mean that games that use lockstep are necessarily client authoritative? Or did I miss something?

I see what you mean. Yes this is exactly that: each client is authoritative, alone in the world - and they just have to trust the other clients will compute the same game.

As with all historic RPG's, The key lesson is in how you craft your C integer types. One type allows you to cleverly assign various character traits to specific groups and users, the other turns Ghandi from a benevolent peace loving icon to a harbinger of endless misery and destruction

https://www.geek.com/games/why-gandhi-is-always-a-warmongeri...

*Gandhi not Ghandi. It is right there in your URL

*Gandhi

It's a pity they aren't releasing on Steam though. Most players aren't going to buy the game from the Win10 store.

> It's a pity they aren't releasing on Steam though. Most players aren't going to buy the game from the Win10 store.

I agree but I'd go even further: it's a pity they aren't releasing as standalone. Where are the good old days when using a locked platform wasn't required to install a game?

It's not the matter of a locked platform per se. Releasing a game on a platform like Steam just increases the likelihood of someone buying it manifold.

The age of physical discs are over, but the developer is still free to release a DRM-free version of the game independently. Witcher 3's Projeckt Red does this. You can simply enter the product key received from Steam to download this DRM free copy.

Those days were gone when DRM became an acceptable thing defended by people even on this site.

Its always that one CEO, wanting to emulate the success that is steam by becoming basically becoming the market and the platform. Self-delusion and believe-shapes-reality needs some realistic observation or else one wastes ridiculous amounts of money reinventing the wheel only for the crowd to flock to the first wheel shop anyway.

Those are resources that could go into inventing the first plane. And some superior sooner or later realizes that you are on a unimaginative crusade and cuts the lifeline, resulting in a dead market-plattform, with contract-bound studios being dragged down with the sinking ship.

Windows Store has been surprisingly fragile for me. I've bought a couple of games through it and things like turning off IPv6 will cause it to fail to download anything and spit out cryptic error codes.

But mostly I don't want my games spread across a dozen different storefronts. Keeping them all in once place is more convenient. I would probably be more open to the Windows Store if I owned an XBox and there was some kind of integration, but I don't.

Steam is doing even worse with IPv6 support – in fact they have absolutely none (although their CDN provider has great support for it). Really a shame that they're missing such an useful piece of functionality in this day and age. This of course has made me consider some of the alternatives...

Unfortunately the only Steam alternatives that have some amount of IPv6 support are Humble Bundle and Green Man Gaming. Though you may as well not count Humble Bundle as accessing their website over IPv6 results in it being completely broken [1].

[1] https://vgy.me/0E4HBm.png

I'm curious -- why would IPv6 access for a website be an important matter?

It's faster, it's cheaper, it's easier to setup and use. As of now not everyone always has IPv4 access anymore. I conciously avoid purchasing products that have no IPv6 support, as I sometimes have an only IPv6 connection.

20% of the world uses it.

Okay but 100% of the world uses IPv4. Why is having IPv6 support important for a website?

Because IPv4 is close to exhausted[1] and we need IPv6 support to be the rule, not the exception, by the time it's no longer affordable to get an IPv4 address.

https://en.wikipedia.org/wiki/IPv4_address_exhaustion

Not supporting IPv6 is better than breaking in mysterious ways when you have IPv6 support but have to disable it on that machine for reasons (Netflix).

I would like to see Valve get with the times on their v6 support though.

I've never seen a dualstack connection cause breakage. And if it does break with IPv6 support being available, then it's the developer's fault. Instead of disabling IPv6 you should alert them of the problem and if possible use an alternative service, which supports IPv6.

In the case of Humble Bundle breaking, it seems simply because some of their CDN subdomains are IPv4-only. Since the site loads various resources from those IPv4-only endpoints, it causes everything to just break when accessing over IPv6. In a way I'd argue it's still better than your browser displaying "no valid IP address for this domain" – usually you can at least find an e-mail address at the bottom of the page to alert them to the problem.

>Though you may as well not count Humble Bundle as accessing their website over IPv6 results in it being completely broken

Just checked this in Firefox and it loads completely fine via IPv6 (and I verified using the devtools that it is using IPv6).

Should've clarified. In this case I was on a IPv6-only connection – no IPv4 access available to the outside. Loading with both IPv6 and IPv4 being available of course results in no problems, as it falls back to IPv4 for resources that are not accessible over IPv6 (which is mainly the layout scripts).

Also, if you have a Windows Update pending that requires a reboot, attempting to install anything from the Windows Store will fail, sometimes with a cryptic error code and sometimes not even that.

I'd say it'a a pity they aren't releasing it on GoG! Win10 and Steam are both DRM.

They can do both. Steam is just a larger platform.

Why not? A MS account is not uncommon.

Personally, I can play a Windows game purchased on Steam on Linux/MacOS using Wine (This is how I currently play Age of Empires II). You can't say the same about a Windows Store game.

There is a stigma in the PC gamer community against Microsoft, especially after Games for Windows. "Steam or go home" sentiments also aren't all that uncommon. As for the accounts - I really don't know anyone who actually uses the MS store (whereas pretty much everyone who plays video games has a Steam account), and I interact with plenty of end-users daily. Maybe it just happens that the area I live in never adopted it, but I would still say that Steam is the better platform to try to get exposure on.

Trading one monopoly for another, I guess.

A thus far benevolent monopoly who have indicated positive intent and applicable thought patterns (i.e. the whole piracy is service problem).

They could release it on origin...

Joke aside, i assume a month later they will release on steam. The red number sirens and the fear of pirates have convinced a lot of sailors to rock the boat.

Can you login to the store without linking your entire computer to that email?

yes, but it's still a bad store :(

Games stores (for better or worse) are expected to have an ecosystem of services of varying importance:

- Friends list

- Text & Audio chat

- Cloud saves

- Automatic updates

- DLC installation

- Sales and Discounts

- Curation, suggestions, and browsing

- Achievements

The Microsoft Store has most of those, more or less, but the experience is not ideal.

The social services are tied into the Xbox app which works okay, but is quite complicated and Microsofty. Also PC gamers might not have their friends on Xbox or maybe it has their CoD friends, not their AoE friends, etc.

The MS store is not purpose-built for games, so it makes it a poor place to buy them. It's hard to find "real games" on there because you have to sift through mobile games, photoshop, movies, and Xbox controllers. Many PC stores make a big deal about sales with purpose-built home pages, best seller lists, weekly sales, event sales, newsletters, etc. The MS store does not have sales often and if they do you're not likely to hear about them. PC stores put a lot of effort into discoverability. There's personalized home pages, followed curators, recommendations from friends, and detailed search systems with a multitude of tags and sorting options. You can search for a recently released co-op action-rpg with steam controller support and sort by recent review sentiment. The MS store, by comparison, is a mobile app store on a computer.

PC Game stores are expected to have fast and reliable downloads. Steam can consistently download faster than my hard drive can write, and can reliably resume interrupted downloads. Steam can validate the integrity of game files and re-download them if necessary. Other clients seem to do fine as well. Download speeds from the MS store are reportedly inconsistent, and it has (or had) no ability to resume interrupted downloads. For weeks after the ~100GB Gears 4 was released exclusively on the MS store, /r/pcgaming was full of people complaining that the download was interrupted at 95GB and had to start over from 0.

All of these problems are solvable. I believe the download issues have been fixed. The store is already the best place to buy "Play Anywhere" titles if you have an Xbox, simply because you get it on both platforms. But MS has a long way to go to make the store pleasant to use and to win back PC gamers' sentiment.

One practically unsolvable problem is that the store doesn't exist on Linux, macOS, or Win 7, which matters to some portion of games and some portion of gamers.

Literally never used the MS Store

I even go so far as to block all requests to MS Store (via winstore.app.exe) with NetLimiter. I also block outlookcommunications, microsoft.photos.exe and video.ui.exe, all of which I never use but they keep wanting to call home every few days).

Why do you stay with a system whose developer has proven to act against your interest?

What's the alternative? And before you offer some obvious answer, remember that we're in the context of a Windows-only game :)

Gaming for Windows Live died. Let it stay dead.

I have never used the windows app store but I might for this, which is probably the point.

Looks like they want to avoid changing gameplay as much as possible.

Considering their focus on preserving even minute details of game dynamics... I wish they'd made an exception for pathfinding - the algorithm in the original AoE is extremely bad.

In particular, I really wish they'd introduce some improved ways to command groups of units. IIRC the Rise of Rome expansion introduced double-click group selection of all similar units, so it's not without precedent. If they change nothing else, my number one wish would be formations like in AoE II. I can't stand the way units just swarm like ants when told to move together.

It's really hard to change something like pathfinding without revamping the entire game. A game's balance is like a hanging mobile. Change one aspect, and the whole thing becomes unbalanced until you adjust all of the other weights in the system.

For example, if you make the pathfinding smarter, then there's a good chance you've made all AI enemies effectively more powerful and now all of the level difficulty is too high. You may also affect the strategy of levels — levels that used to have carefully crafted chokepoints that the player must notice may no longer be chokepoints with smarter pathfinding. That in turn might lead you to want to tweak the design of the level...

Except this is a RTS and when players want a unit to get to a place it should get there without taking a detour across half of the map, especially when the player can visually see an optimal path.

So no, there are no downsides to better pathfinding.

consider the following scenario:

In a battle between melee and ranged units the ranged units want to maximize their surface area touching the ranged units so that all the melee units can attack at once and quickly change targets once an enemy unit has fallen. The ranged units want to minimize their surface area so that only the outer layer of the group will be attacked while every unit on the inside can freely fire on the melee units.

In an older RTS with poor pathing this meant constantly jockeying your forces around trying to either get a good surround or keeping rank. Moving your forces in certain ways could unpredictably change the formation of your units so great care had to be taken when engaging the enemy. Getting this to work out was a skill in of itself that took strategy, positioning, the ability to predict the enemies move, etc. All of this depth could take place between very small groups of very simple units.

In an RTS with perfect pathing the ranged units always move in a perfect sphere (minimizing surface area and maximizing volume) and the melee units perfectly synchronize their movements so they don't trip over each other when surrounding the enemy. Once it is decided that a group of simple melee units and ranged units will fight there really isn't much for the player to do except tell them to get on with it.

As real time strategy games have evolved along with better pathfinding, we've found out that with all things being equal, perfect pathing makes for very very boring fights. Active abilities had to be added to give the player something to do during the fight. Rock Paper Scissors type unit counters had to be added so that positioning mattered again. More AOE spells and debuffs had to be added so that another thing would force players to jockey units around.

Study the design of Starcraft BW and Starcraft II and you can see this very clearly. Certain fundamental unit matchups were completely broken by improved pathfinding such that more and more complicated mechanics had to be added to the game to give them balance and dynamism again.

Of course, this only really applies to C&C, SC, and AOE style RTS games. More modern RTS series like Men of War and Wargame don't follow this pattern because the range and lethality of the units typically far exceed their movement speeds and terrain and cover play a much larger role, but that is a whole different topic.

It can break the whole game balance, have you ever read about Starcraft and how its wonkey pathfinding was a big part of why it was a esports success (allegedly)?

> Assume the original developers knew what they were doing. The old code shipped and was successful. If you don't understand it, most likely the problem is you, not the code.

This needs to be chiseled on a tablet somewhere.

Really great piece of advice. The "every other developer is an idiot" mentality that can be popular in some people/cultures will almost always bite one in the ass.

In general, I liked how applicable a lot of the points made in this post apply to development out side of problem space videogames.

Love reading gamedev post-mortems such as these ;)

Although what author discusses is specific to Age1 engine. Time sync in webrtc peer networks encounters many of the same pitfalls. With errors tending to propagate non-linearly as more peers (and more complex scene objects) are added.

ThreeNetwork - Network sync library for Three.js

https://github.com/takahirox/ThreeNetwork

We got way better at dealing with synced/deterministic simulations over time. Much better sync logging system, decoupling render rate from simulation rate, more institutional knowledge/intuition about all the surefire ways to break determinism (using unsynced random number generator, uninitialized variables, reading local machine state from sim, using local timers rather than synced ones, not resetting floating point flags after calling Direct3D, etc). The bad news was once we were generally avoiding all the common causes as a matter of course, the sync bugs that remained were non-trivial ones that took lots of time to diagnose.

I'm surprised they decided to modify the old code, I expected them to make a clean start and re-create the original experience from there.

Way before Counter-Strike: Global Offensive people tried to port the classic CS 1.6 (or 1.5) behavior to more modern engines or CS versions. From what I know no project succeeded because even replicating the recoil behavior were extremely hard. Gameplay features that were actually bugs would have to be simulated. I imagine it would be ten times harder for a complex game like AoE.

You mean like Netscape did?

I guess it would have been quite hard to do if they wanted the same gameplay and not spending too much time on the balance between civs and units.

They tell for the network and pathfinding code than redoing would have been stupid as it has already been done and was working. I guess it is the same for most of the system/gameplay code.

I'd equate it to the difference between restoring an old film, vs re-filming it with new actors/director etc...

Not building custom containers is generally good advice. The chance you build something that underperforms is noticeable. Other coders won't know what you've made. There is a fair chance that std has what you need if you're even just a little bit thoughtful.

Note that the stl containers were far from perfect a few decades ago.

Amazing what a few decades of development by talented and determined people can accomplish. Point is, my custom container that I wrote in an hour or so probably won't be as good.

Fun read. I've always wondered what kind of experience it takes to get picked for a project like this. Seems like it would be fun.

Cool blog post, but light gray text on white is literally unreadable. How are people still doing this?

Good thing they decided to keep the gameplay the same, like Starcraft: Remastered. I never played AoE 1 competitively, but if it's like SC many bugs/glitches became core gameplay mechanics. Here's hoping the AoE 1 remake fares better than the one for AoE 2!

I have no real experience with C/C++ as well as game development but I always wonder what would happen if old games like these would get recompiled with an, for example, avx2 flag.

Very little. Quite a few programs recieves a decent boost when compilers moved from a 386 default to 486, then Pentium 1. Since then the gains have been much smaller, unless you use the instructions on purpose.

Reply on news.ycombinator.com