I cut GTA Online loading times by 70% (2021)
nee.lv
nee.lv
One thing to note - assumption that one might make is that "Rockstar shipped obviously ridiculously suboptimal code; how did this ever get past testing??"
However, based on the author's discussion of the code, and the issue being in inefficient parsing of an online store content, it is likely that this not only passed testing, but was sufficiently efficient in production too... upon go-live, when the store content database was small.
This likely grew slowly, and was especially bad for a specific subset of users (CPUs with poor single core performance). They may even have had trouble replicating it in non-production environments depending on their refresh policy and whatnot.
This is not to excuse it, of course; but I've seen a repeated pattern where a not-perfectly-optimal algorithm works well enough in development, testing, and initial production... then blows up in production with growth a couple of years later.
(In particular, I deal with a lot of complex SQL in my day job, and they have a nasty habit of going exponential with data growth. If you're implementing a greenfield application with no objective knowledge of its future size and growth patterns, it may be littered with trapmines despite reasonably good efforts to not make it so. Especially dangerous with "Big Bang Go-Lives" - though a game may have internal Agile sprints, I'd still consider it overall a bing-bang go-live in that majority of its content will go live at the same time, and then you get to see how your 100s/1000s/10,000s of SQL/code/libraries/etc behave:)
There was also server instability near launch which it may have been thought to be linked, as in we thought we were waiting 10 minutes for a server. But any actual server issue was probably more demand/load scaling based.
it was obviously not network based as connection speed didn't seem to change the time required
"Scrum", "Agile", "Lean".
The devs were may have been too busy "sprinting" and getting their assigned tickets done to think about higher level concerns like "is this good? can we make things faster or better?".
I think your points also stand though.
It's also completely baffling that a company would invest such resources building such a complex game and be fine with 6 minute load times, caused by such ... dumbness. Like selling a sportscar with a 5 litre petrol tank.
They are probably just adding more content, but the original dev's algorithm for loading that content didn't expect a bazillion new cosmetics
But I can understand thats its normal large companies are so incompetent, and at an older age I appreciate it gives a chance to the little guy or company to rise up, for now, until AI starts to catch these issues and large inefficient corporations can finally solidify their place in the hierarchy.
The problem was sscanf calling strlen. You're going to fire your management and development team because one dev used sscanf where it's probably appropriate?
Then hire a guy who may or may not have actual game development experience and give him $100k because he took who knows how many hours to trace through and find this thing. The guy himself the fix would probably take a day to implement.
Finding it could have taken longer.
You want to call Rockstar incompetent, but you're simply going to ignore everything else that goes right in not only GTA Online, but across all of their games.
I'm sorry, no company is going to pay someone only to arbitrarily profile their software looking for potential bottlenecks to remove.
The only reason this story is notable is that there was a conclusion. He could have easily found nothing to fix.
It's just a stupid, knee-jerk reaction to call for the heads of everyone remotely responsible and replace them with the guy who happened to find the issue. And of course it looks like a straight shot from noticing the problem to finding the problem. I'm not going to put in every single dead end. Unless they happen to be interesting or relevant.
While the past looks like it obviously leads to the present, it only does that because we've already walked the path.
And what is that guy going to do the rest of the time?
That’s classic pre-devops dilemma. Let’s fire all these slackers and get screwed next week. How about nothing to do means they’re doing an excellent job.
But in general, given this idiotic reality, you’re probably right. I just don’t adore it as much.
It's probably those sort of reactions that got Jobs booted from Apple the first time.
This is kind of like hiring someone who won the lottery to be your financial advisor.
Just for the record your swearing makes you uncivilised. I was just offering my opinion. I’m allowed to do that. I was giving my professional opinion. All your replies not only make you look like you can’t take criticism but they are also IMO mostly poor and short sighted. However I appreciate your time and input and will try and reflect on them. Have a good day man and for what it’s worth Rockstar has always been one of my favourite companies over the past 2 decades.
My opinion is if I was management in that company I would accept my fate if such a transgression occurred on my watch. I wouldn’t be butthurt about people thinking I should be fired.
Ten people wasting 6 minutes doesn't necessarily waste an hour. No one is curing cancer with the time "saved" from loading GTA Online. More likely than not, they're playing GTA Online 6 minutes less.
And to be fair to Rockstar, which the author was, this problem lurking in the standard library isn't really to be expected. You expect your standard libraries to be decently optimized.
And why it was undiscovered for years has a very simple answer: It wasn't a priority.
Why should that problem be prioritized over everything else? Everything else that could "burn lifetimes away"? Everything that could make the game crash? Or cause people to stop playing? It's quite possible that this issue was always slightly less important than whatever else may have come up.
That's an assertion. One I don't accept without some strong backing. The extra 5 minutes might mean a discussion somewhere is a bit longer leading to valuable insight. It might mean just trying that last thing before finishing up which you wouldn't try because it shouldn't work anyway. You can imagine any world changing situation which only occurs because of those extra few minutes.
> Why should that problem be prioritized over everything else?
Because it's burning away lifetimes. Besides no one would have complained if this was in the game for a few months and then fixed. It was in the game for YEARS. That is not "prioritzed over everything else" that is "the priority is so low you have to dive to the mariana trench to find the ticket".
I need to see some data on this.
I am baffled nobody at R* cared to investigate a 6min load time that was there for years. Doesn't sound very AAA-gamedev to me.
They're absolutely jacked workstations that probably loaded it in a blazing 2.7 minutes!
If there's a million fires and half of those crash the game, fixing something like a load time might easily stay under the radar.
And that's even before we consider internal political issues. The code could have been owned by a group that was refusing any PRs that would have fixed the issue. Or maybe the code was written by the big boss' nephew and fixing it would have removed his contribution.
Don't underestimate the power of complacency
If you extend the logic, what's the point of keeping any commits from years ago? What's a good cut off date to just start discarding history? Rewriting it?
But in the same breath, what's the point of cloning commits from years ago? It's not as if you'd cherry pick anything from back then. And that's my point, the former requires an org decision and active action, and the other is a blobless clone. The nice thing about the blobless clone is you can always change your mind, you can get old blobs if you find you need it. But if you threw out old commits, and you need them later... that's the brakes.
1. It's a boiling frog problem which developed over several years. It started fast when the JSON file was small but then got a little bit slower each time new content is added.
2. It's also not exactly an "interesting" part of the code base. IIRC it's a JSON file which configures the ingame shop(?). This was probably delegated to a summer intern to implement in a couple of days, maybe in an entirely different department from the people who develop the actual core game.
3. New shop items are probably added by a content team without programmers in the loop who would know by instinct that something must be wrong. People on that content team also most likely don't stay on that team for very long, which together with the boiling frog problem above means that everybody thinks "it's always been like that".
4. Any specific feedback from the outside probably gets lost in the noise of other requests.
5. ...plus a general 'not my problem' attitude common in big organizations and an always-full kanban board ;)
My theory is the fix existed internally but was never merged for weird political reasons. Some have guessed it was due to advertising being done on the loading screen. Or it could have been pure dysfunction, like how fixes for all the notepad.exe problems apparently existed at Microsoft for decades, but it took a full rewrite to have working undo.
The 'large-organization-dynamics' are all the same though across all industries. I bet enough people down in the trenches noticed, but nobody pushed enough to get the problem investigated and fixed. There's probably several dozen bug duplicates about that problem buried somewhere in GTA5's JIRA (and indeed probably also with exact instructions how to fix, and maybe even somebody did that work but then didn't push enough to get it integrated).
I work in a larger business but am dealing with a similar problem. Trivial implementation/fix; nearing two years into the effort.
What am I doing? Changing 'proxy.remoteURL' for our Docker registries from a painfully slow upstream to a faster one.
You can have people in pain with a solution in hand and still find it difficult to sell.
> You can have people in pain with a solution in hand and still find it difficult to sell.
"Well, we don't know what side effects that might have"
Actually, my understanding is that most devs don't really play the games they build, at least not in the way that people do. (Anecdotal evidence is that quality-of-life features tend not to come from developers very often). And even when they do play the games, internally, it's likely on an internal branch, with internal mocking of things like servers and the like, so a loading time bug that scales quadratically with content is unlikely to be something that would surface in internal playtesting. I also doubt they're playing their games on their own time; if you're already working 40 hours or so each week on the game, it's probably not a feeling of unwinding to put in another 5 or 6 hours on your own time after you come home from work.
My guess is that people noticed but nobody was in the right position to debug - their internal tooling may have been that this code path was usually skipped or behaved differently on internal builds. And debugging a full production build was something many devs didn't do because it was painful.
6 Minutes?!?!
Are we using Commodore 1541 FDDs again?
Per the article, most of that wait is garbage coding and that's apparently just fine, because....clowns will continue to buy it.
They won't be. They will push out worse and worse code and it's fine, because it's still being purchased.
Have you heard of a game called Fallout 76? or maybe Red Dead Redemption 2?
Both released as hot garbage and apparently, that was fine.
"You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day."
"Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"
Even if you give them an extra 10 seconds every day. You would need to give them an extra 10 seconds several times throughout the day.
Time doesn't really aggregate in people like other things.
By learn to use I mean really do, it's like refactoring tools, it's sort of boring and tiring to learn those, you end skipping large parts because "yeah I already know / it's obvious", then you need one day for an emergency / something right now and feel like you're fighting against it.
For those three (and some others), the key to being very good at it is learning all the small tricks and details, when you don't need it, so the day you do it lets you skip 90% of the work.
As an overall point, I'm sure many people have various different experiences, but for myself while being in high school I needed to parse "replay" files for some games to extract and process information and allow players to post their replays to get stats on web forums, and it helped me to learn a great deal about how to properly reverse engineer and figure out stuff, things that I have no idea how I would have learned otherwise.
Personally I have experience with JetBrains ReSharper for (full) Visual Studio, it has loads of functionality which makes it very fast for me to navigate and refactor even very large C# projects.
JetBrains has different products for many different languages and I expect they offer similar capabilities in each.
Learning all the features at once is impractical, but if you learn a new one every now and again, after a few months or years you become incredibly proficient.
I had a similar experience with conversation macros back in the day when I was a GM for WoW.
You don’t make too many at once as you’ll never remember them all, instead you make them kind of as you need them, which tends to be situations like “this is the third time today I’m writing this long explanation”.
I mostly build free-cameras, which allows you to detach the camera from characters, and for that I had to learn x86 assembly, some disassembler tools and how compilers generally work.
I think it's super rewarding because while you're seeing bits and bytes, and crashes, most of the day, at some point you can see you achieved something by being able to freely move in a restricted world.
I was surprised how far you can get with Cheat Engine. Its disassembler is really good, and you can do injections directly, if you then want to move to your own advanced stuff, you can build frameworks to do all of that using Windows APIs.
Don't jump in with "hi please teach me" or asking questions in the FAQ/wiki or found easily on the web.
Try to give back to the community (like fixing/improving the FAQ/Wiki) as early as possible.
Windows provides _snscanf_s if you want to keep track of the string length yourself instead of having it recompute it each time.
>To be fair I had no idea most sscanf implementations called strlen so I can’t blame the developer who wrote this. I would assume it just scanned byte by byte and could stop on a NULL.
The author's replacement strlen does the "cache the length across calls" thing only because bolting that on top of the default strlen was easier than doing the lazy parsing thing, since the latter would've required making an actual sscanf implementation to do that from scratch.
(?P<NUMBER>?\d+(\.\d*)?)|(?P<ASSIGN>:=)|(?P<SEMI>;)|(?P<ID>[A-Za-z_][A-Za-z0-9_]*)|(?P<ARITH>[-+*/])|(?P<NEWLINE>\n)|(?P<WHITESPACE>[ \t\r]+)|(?P<MISMATCH>.)
Of course, there are implementations whose authors thought about that and decided to do the reasonable thing instead, e.g. musl [2] and Plauger's old stdlib [3].
[0] https://github.com/lattera/freebsd/blob/master/lib/libc/stdi...
[1] https://github.com/bminor/glibc/blob/dff8da6b3e89b986bb7f6b1... , https://github.com/bminor/glibc/blob/dff8da6b3e89b986bb7f6b1... , https://github.com/bminor/glibc/blob/dff8da6b3e89b986bb7f6b1... — two additional files and five times as much code compared to FreeBSD's implementation for pretty much the same functionality, wow.
[2] https://git.musl-libc.org/cgit/musl/tree/src/stdio/vsscanf.c
[3] https://github.com/wuzhouhui/c_standard_lib/blob/master/STDI...
I can't "prove" that, because the data isn't there, but it is a completely reasonable guess. At an 8 minute load time it is reasonable to guess there were a lot of people who stopped playing and thus stopped spending because they thought their install was crashed or corrupted, and just silently wandered away. I certainly wouldn't. I've got some games that load more slowly than others but I don't think I've waited multiple minutes since the Commodore 64 era.
- Luke Stackwalker: For stack sampling (perf analysis)
- ???: industry-standard disassembler
- Process Dump: Dump process memory to file (for obfuscated code analysis)
- x64dbg: For debug stepping.
- MinHook: For patching/modding a binary executable.
guess that's IDA Pro
It is an interesting case in piracy, as being the go-to software for reverse engineers, it also has the biggest target on its back for software crackers. Despite this, newer versions are not always cracked and released publicly so quickly. It seems each license owner gets their own custom watermarked build of the software. The crack of version 7.7 was leaked from Think-Cell, the excel add-ons company. 8.3 was recently seen released, with an anonymous supplier and Vietnamese cracking group behind it.
I cut GTA Online loading times (2021) (June 9, 2022 — 476 points, 213 comments) https://news.ycombinator.com/item?id=31681515
How I cut GTA Online loading times by 70% (February 28, 2021 — 3883 points, 699 comments) https://news.ycombinator.com/item?id=26296339
Played recently and thought Rockstar must not have actioned the advice in the blog
But no it's just it is still slow but they did fix these issues kudos to them but performance still needs attention
People who play this regularly have their phones out/youtube videos on waiting for these load times
Most PC gamers play multiplayer mods like RageMP that allow connecting to servers with well-made scripted gamemodes like RP, but also stuff like death match where hackers are actually banned I'd guess.
this really makes me think:
if this obvious flaw exists ... imagine what other, more complicated and opaque mistakes exists and we're not aware of it ...
just food for thought