Hacker News Reader

The big difference is in the net code. While I can remember and justify all of my decisions about networking from DOOM through Quake, the bottom line is that I was working with the wrong basic assumptions for doing a good internet game. My original design was targeted at <200ms connection latencies. People that have a digital connection to the internet through a good provider get a pretty good game experience. Unfortunately, 99% of the world gets on with a slip or ppp connection over a modem, often through a crappy overcrowded ISP. This gives 300+ ms latencies, minimum. Client. User's modem. ISP's modem. Server. ISP's modem. User's modem. Client. God, that sucks.

Ok, I made a bad call. I have a T1 to my house, so I just wasn't familliar with PPP life. I'm adressing it now.

The first move was to scrap the current net code. It was based on a reliable stream as its original primitive (way back in qtest), then was retrofited to have an unreliable sideband to make internet play feasable. It was a big mess, so I took it out and shot it. The new code has the unreliable packet as its basic primitive, and all the complexities that entails is now visible to the main code instead of hidden under the net api. This is A Good Thing. Goodbye phantom unconnected players, messages not getting through, etc

It was Carmack - I've always loved this article https://fabiensanglard.net/quakeSource/johnc-log.aug.htm

Yep, and because of his .plan files, and no other historical details around the technology, I think it's safe to say he invented it. In 1999, QuakeWorld was the first, to my knowledge, implementation of client-side prediction in a first-person shooter.

But I'm almost certain there were rudimentary client-side prediction models that existed before John Carmack, which is why I say I think he was the first to "invent" it, but frankly I'm not so sure about that.

I mean think about terminals, as an example.

I do think with near certainty he was the first to invent the client-server with client-side prediction architecture in time stepped software, or more specifically in an FPS.

It's indeed sad to see. Game networking techniques have been known for multiple decades and yet still haven't made their into any of the major publicly available engines (notably Unreal and Unity). What gives?

As a game dev of 10+ years myself, I have a few theories:

1. Multiplayer networking is the "secret sauce" that creates moats/barriers to entry for incumbent studios. Think Rocket League or Fortnite or StarCraft 2. The tech exists, it's functionally well understood, but hard to implement at a high level of quality. Why not keep it to yourself? There is more money in holding onto your game revenue monopoly than trying to sell the network tech.

2. Game genres differ greatly in their networking needs. Rollback is great for fighting games but that's about it. StarCraft uses delay based deterministic lockstep. FPS games use (typically) ad-hoc server authoritative state syncs with client side prediction. Other complex games use full determinism with rollback, sending only inputs over the wire. Some cheapo indie games using client authoritative models (open to cheating but easy to implement, arguably fine for coop games). There are even more variants and blended approaches but you get the picture: there is no one size fits all approach and many are mutually exclusive so it's harder to package into an engine as a comprehensive solution.

3. Networking a game properly involves very leaky abstractions. It is impossible to write gameplay code for a networked game without understanding the nuance of the network model. This makes it substantially harder to develop the game, and this hurts the major game engines' marketability, with both major players Unity and Unreal guilty of selling themselves as "look ma no code required" solutions. Similar to point 1, not worth the money to sell this.

I don't see a great way out of this unfortunately. The only good networking middleware I know of is Photon and they're not exactly an easy to use product either. Hopefully we see better open source tooling in the future.

I agree. Sell the physically based rendering sizzle, keep the multiplayer networking sauce for yourself.

I don't buy the different genres differ greatly in their networking needs, though. Dota 2 uses the same networking architecture as Counter-Strike. Both of those games use the same underlying networking techniques as RuneScape 2 (Old School Runescape). Those are a MOBA, FPS, and an MMORPG respectively.

You use lag compensation (now people refer to it as rollback) in FPS games too. Yahn famously talked about it in his paper.

In fact, any type of granular timing-dependent gameplay requires lag compensation, otherwise you end up with situations like players leading their shots or missing entirely (Halo 1 PC).

In Halo 1 PC, they actually broadcast a hit sound from the server if you actually did make bullet contact. And those weren't trace-based (hitscan) either. Halo 1 bullets actually had travel time. So, Bungie basically didn't do lag compensation correctly, and hacked a solution on top that gave players feedback to help smooth over multiplayer. Because bullets had travel time, too, it just felt natural. But make no mistake. It was still wrong, because when bullets landed on your client, they didn't land on your target on the server. You might have hit the player on the server, heard the feedback, but saw dirt fly up from the ground.

Basically, all real-time games have the same networking requirements.

If you made a 2D chess game today, you'd still have a game loop and networking would be a part of it, sending payloads probably only when players took their turns. That same basic design applies to FPS games.

Chess doesn't need client-side prediction, but as soon as you want to allow other players to see you moving pieces around based on cursor position, you're sending real-time data by payload over a time stepped game loop.

You just don't need to predict anything, nor do you need to interpolate or anything else.

> It makes the whole engine feel like a joke.

1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider...

I've tried on many occasions to get excited about these engines and started a project with them. Unfortunately, when the entire realm of multiplayer feels half-assed my hopes are dashed catastrophically.

Right now, I am spending every moment of my free time developing my own game/engine from scratch. I honestly don't give a shit about Nanite, AAA graphics, et. al. Complex multiplayer experiences are far more compelling to me than raytraced graphics and sub-pixel triangles.

I think the exact nature of the net code is much more intimately linked to the game concept than anything in the rendering pipeline (for a multiplayer experience like Quake or Counterstrike). You will never get a perfect fit if you outsource this.

Yes! I completely agree. You're not alone. There's a lot of us out here slogging through writing portions of game software we don't want to only because it's necessary to create the experiences we want that can't be delivered in a timely fashion with existing technology.

> 1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider...

Intriguing. Makes even more sense that Frost Giant Studios (who are using UE5 for their upcoming RTS) have written their own networking code (with rollback!), and thrown away other big parts of the engine.

I'd be very interested in reading more about FGS and what they've been up to around UE5.

The idea of incrementally defenestrating UE5 in favor of in-house components is compelling to me. This path definitely works a LOT better in a team setting where you have actual artists looking for things to work on.

I don't know how the licensing actually works out around this though... Assume you went full ship of Theseus on UE5 and replaced 100% of it with in-house. Will Epic Games be happy walking away from this, or is there some fundamental legal component that we know we'd never screw with?

You can modify Unreal Engine as much as you like to ship a game and the engine changes remain your property, you just have to pay their royalties.

At least that is how it is in practice - the language in the license agreement is actually a little fuzzy, when our game was sold from one company to our current one, the lawyers had a bit of a frowning session over some of the language there, but ultimately decided it was okay. Maybe in the future it will be challenged, but for now, self-owned engine changes are the norm.

> I'd be very interested in reading more about FGS and what they've been up to around UE5.

They have two subreddits: r/frostgiant, and r/stormgate (for their game). They regularly ask the community for feedback/discussion/brainstorming on some topics relevant to the gameplay (like pacing, asymmetry, monetisation, e-sports, etc), openly share their conclusions from the discussion, but unfortunately in-depth technical stuff doesn't really pop up often.

They specifically bragged about their pathfinding (apparently at least on par with StarCraft II), and the rollback netcode (which is "directly inspired" by fighting games, and apparently not a common thing for RTSs), but again, everything is scarce on technical details.

The unofficial Discord community (https://discord.gg/stormgate) has a "Yeti Tracker" channel that automatically cross-posts any comments made by FGS staff; there's also https://stormgatenexus.com (fan website) where many of the interviews are transcribed and distilled into articles.

I'm watching all of these closely, I can ping you if I see more engine-tech-heavy stuff popping up!

It's not because they wanted to, I guarantee it. It's because it's impossible to not do it. Critically important details, down to the engine's frame timing, how replication is deeply integrated into the engine, and how the default Actor movement code works means you have to not gut just a single implementation detail, but you have to cut whole swaths of code out of use.

I'm dying to be proven wrong, but the intersection of people who know multiplayer networking well, Unreal Engine 4 or 5, and have successfully implemented client-side prediction with reconciliation and rollback in said software and out of it as well, are vanishingly small.

I failed to understand how one could use the existing engine's loop which is variable framerate, the existing replication system, which favors an implementation opaque RPC system over explicit payload structures, and rewrite Actor movement and engine timing subsystems without having a parallel network loop that uses none of the above.

My guess is that it is possible to use Unreal's variable framerate and replication system, but only rewrite Actor movement implementation details by creating a class that inherits from it and overrides the behavior entirely, but you're still stuck with the fact that because the Unreal Engine uses a variable framerate, you have an inconsistent rollback timeline across every client.

You want consistent frame times, in my experience, to have rollback. Does it mean that it's impossible otherwise? No, but you can get caught rolling back in-between game states that don't actually exist on the granularity of the timestep you're running at.

It is certainly doable, there is a plug-in on the UE marketplace that meaningfully extends Unreal's movement replication to do nicer interpolation and extrapolation of movement, it is called SmoothSync. We used it in lieu of UE's character movement component on the vehicles in the game we shipped a couple years back. It is a bit heavier on bandwidth (to their credit, Unreal's movement component replication is VERY lean) but still well within budget.

The code is horribly ugly, but works well. The ugliness I think is less to do with integrating with Unreal and more to do with the author though. That said, it works.

I'll have to check this out, thanks for mentioning it. I would be thoroughly surprised if it actually integrated with the rest of the engine though--my guess is they sacrificed this deep integration (foregoing the replication replay system, etc.) just to get something that works.

Pardon my ignorance (only interested in gamedev tangentially), but I would've assumed every game engine that progressed beyond a mere toy would always fix the timestep (or at least make it a single API call to do so). Or is there an entire class of games that somehow benefits from variable timesteps?...

You're not ignorant, it's an excellent question.

The Unreal Engine has a long history, and its origins did not start from a fixed or semi-fixed timestep. The Unreal Engine doesn't benefit from a variable timestep, it's hampered by it, and the engineers have deemed it too difficult to fix, because the decision permeates the entire engine.

There is a "fixed timestep" mode, but it's for cinematics, and specifically for synchronizing the engine with external hardware, IIRC. But it's not a true fixed timestep, it's actually a fixed framerate, instead.

I know I'm going against the grain and this is a super-unpopular opinion, but I really feel like there's room for more specialized engines and more independent engine middleware/tooling in 2023. The "game engines are impossibly hard, so we won't get new ones" discussion only ever talks about Unreal and Unity, where editor, engine and technology are all vertically integrated and the flexibility requires a lot of compromises. Plus you can't really replicate those two, even if you have a great team and 10 years of work on your hands (Godot). This oligopoly is super unhealthy for the industry, IMO.

I don't think it's unpopular. We need more and better game software, what exists today is unsatisfactory for a plethora of reasons. I'm personally interested in this because I ship Planimeter Game Engine 2D, which comes with client-side prediction standard, and it's the largest pure-Lua game engine on GitHub.[1]

[1]: https://github.com/Planimeter/game-engine-2d

I think in most game development related forums, tutorials, etc, you will be correct about the unpopularity of building a new game engine. However, it’s definitely not a lost cause in general. I also agree that a vast majority of the ‘engines are impossible’ are predicated on Unreal/Unity as a baseline. I look at Arch Systemworks and see them using Unreal for their heavily stylized games (Guilty Gear, Drgaonball FighterZ, etc) and watch the various talks and articles about them having to bend over backward and work to fit Unreal’s expectations to make the engine render and animate in the manner they require and always wonder if a custom engine would not make workflow and maintenance simpler. I wonder the same for more specific genres, like RTS and HD2D-style JRPGs, where a custom engine would conceivably be a better long term option for developers. I always go back to Mike Acton talking at the first HandmadeCon, where he is talking about small dev teams and their benefits in specializing tools and wonder if that logic should be applied to engines as a whole.

Exactly! Good point about forums/tutorials/courses pushing unification.

And interesting, Mike Acton's point goes all the way back to the late Fred Brooks. Not the popular main topics of MMM, but the "Surgical Team" chapter, where he mentions that there should be someone taking care of the tooling. Now, in 1974 this could have been a person. In the 90s game studios it was common to have specialized teams building reusable tooling. This seems to be disappearing.

If I had to guess it becomes a question of how easily can you replace/train up new people to be proficient with the engine/maintain it. Fighting game engines are probably the simplest game engines there are to make but new people still need to be onboarded on it, the knowledge will be internal mostly, you can't find people with knowledge in $propietary_engine to hire right off the bat let alone maintain it. Capcom will use I believe their RE engine for sf6, already used it in dmc5 replacing MT Framework so it's not like it isn't happening in some places, but the AA developers where most of fighting games sit probably can't justify the costs to train and maintain a proprietary engine versus the cost of dealing with roadblocks in UE.

I definitely agree.

My original point, that was not well articulated at all, was more that there should be a middle ground. The choice shouldn't be only between "homegrown engine" and "Unreal/Unity oligopoly".

I'm saying there should be much more competition when it comes to third-party engines, but also in third-party tooling and third-party middleware.

And this competition should be more specialized in order to demand less of the "train and maintain a proprietary engine" but also less of the "dealing with roadblocks in UE".

HD2D games are made in UE4 and they seem okay with it.

(A lot of people think it's a custom engine though.)

I’ve wondered how much of this comes from the huge amount of game programmers that dive right in with zero CS background. I’m not knocking just jumping in, but a game engine is definitely not going to be feasible with 0 experience/background. Even setting up a sophisticated enough build environment to glue together libraries and handle assets is not going to be realistic for someone who’s only experience is installing mods and maybe an OOP course.

Oh, I'm not talking about regular game developers making engines from scratch, I'm talking about the need for competition to Unity and Unreal from other companies, either in the form of new engines or new tooling. I believe new, more specialized, engines would actually make things easier for newbies.

But there's a point there, lots of recent game developers are strongly against the idea of "new" things since each new engine would be one new software and platform to learn.

> this is a super-unpopular opinion

The adverse reactions I occasionally receive when I talk about building my own engine are a primary source of my motivation these days.

Let me give you some even-more-unpopular opinions. I think the GPU is a gigantic distraction from fun. Low-level languages are largely a mistake for new engine/game development in 2023+. You can deliver production-quality, interactive experiences on top of GC'd languages that perform better than ASM/C/C++ implementations (both for technical reasons & human reasons).

For me, the best "game engine" on earth right now is .NET7. Nothing gives me as much control over such a broad scope of fundamental primitives. You might be balking at this point, but go look at their System.Numerics implementation. First-party, SIMD-enabled code for projection matrix calculations based upon FPS-style semantics is just the beginning. No royalty bullshit either - I get to keep 100% of the profits I extract.

I should tell you as the publisher of a large game engine that performance absolutely does matter and it will eat you alive after it is too late and you've invested too many years into a particular language ecosystem.

Performance matters but to what extent?

I am not looking to support building WoW2 or COD. This is for a very specific creative vision.

To be slightly fair to the fighting game community, this article is from 2019. My understanding is that most top-line fighting games now are rollback enabled (the pandemic driven shift to online tournies put significant pressure on developers).

My history is a little hazy but wasn't rollback as a solution driven by the community. Maybe it was SF3 online emulation or something? Or was it Smash Melee

At least 2006 https://en.m.wikipedia.org/wiki/GGPO

Ah yes this is what I had in mind

Rollback was added by the community to Melee. With in-game matchmaking. It's a combination of a modded Dolphin + cheats to mod the ISO itself.

Sounds about right. Never was part of the Melee community but I was always impressed at their ingenuity to figure out multiplayer in the face of Nintendo's active community hostility and bonkers multiplayer UX

No, the first major game to use it and push it was Skullgirls.

An exemplary example of how to make a great game. Unfortunately it has been though a series of unfortunate events and never gets the recognition it deserves

"Series of unfortunate events" is an understatement, but I would say it gets due recognition--it's an indie game over a decade old and still earns its slot at major fighting game tournaments and (after said events' hopeful end) still has support from developers and community alike.

Looks like this was published shortly after GGPO was made open source in October 2019, along with a big wave of similar rollback hype. It has indeed become more commonplace in the last 3 years.

Rollback hasn’t been in fighting games for very long at all. It’s a relatively recent addition to the genre, and has resulted in dramatically improved online play. It’s still new enough that it’s even touted as a feature & selling point in marketing. Within the fighting game community, online play/tournaments have typically been regarded as a joke & not representative of skill at all because delay-based networking woes. However nowadays, with rollback, online tournaments are held regularly, more accessible than ever, and regarded more highly than before. Rollback is greatly helping the genre stay alive.

Fighting games have had rollback netcode since at least SF2 HD Remix in 2008. In terms of online fighting games, it has been around for more than half of their existence.

The problem has been the lack of adoption/prioritization, and probably more than a little NotInventedHere-syndrome.

One of my biggest dings against Godot is not having support for Photon Networking's newest products

Photon has a new group of network libraries that are essentially the evolution of "rubber band networking" that's been default for networking middleware for the last decade

One is especially interesting in that it uses IL weaving to provide a completely deterministic engine for simulating gameplay based on inputs only: https://www.photonengine.com/quantum#

Determinism from inputs was basically a part of QuakeWorld, and how the "VCR" system in id Tech 1 worked. You could replay sessions because the inputs were what were networked, not other state. They're selling 24-year-old tech. :)

If you can make 24 year old tech turnkey, you have a business

Yes! I think this is proven very much by the industry widely still being far behind the curve of what is possible.

Is there any good template/library for modern multiplayer netcode for Unreal or Unity or anything else, or is it truly everybody rolls their own as the secret sauce in their AAA games?

No, people literally still roll their own multiplayer networking in 2023. Seriously. Unreal Engine 5's out-of-the-box multiplayer code isn't production ready, and they're probably the "best" of the major engines people use today.

It pales in comparison to older engines from other game companies. It's effectively unusable.

This seems wild in the free-to-play era, where multiplayer with in game purchases is, like, the main business model. Maybe Unreal figures the multiplayer implementation is the metric on which the game studios compete, so there’s no hope to commoditizing it, or something like that?

It's also why so many multiplayer games are bad. Wanna know what's worse? Unreal Engine's architecture is unreliable by design.[1]

Basically, Tim Sweeney's approach for Actor movement wasn't to replicate state accurately, but just trust that clients would eventually get the right data. This isn't the same as eventual consistency mind you, because game state constantly changes in fast-paced games.

You can have whole time spans where Actors have absolutely no accurate player position replication.

[1]: https://docs.google.com/document/d/1KGLbEfHsWANTTgUqfK6rkpFY...