Rockstar thanks GTA Online player who fixed load times, official update coming
pcgamer.com
pcgamer.com
Rockstar awarded them 10k for it.
How I cut GTA Online loading times by 70% - https://news.ycombinator.com/item?id=26296339 - Feb 2021 (697 comments)
On one side, yeah - the work he did will be worth a lot more than $10k to Rockstar. They've had lots of people complain about & stop playing GTA online because of the slow loading times. And this issue would probably never be found or fixed otherwise.
On the other hand, he did it for fun (without any expectation of reward). $10k isn't massive and we can quibble over the reward amount, but its delightful that companies are rewarding people at all for fixes like this. A few years ago he'd be lucky to get sent a GTA t-shirt. Its definitely the right direction we should be moving in - bug bounties are a thing in security, and we should be celebrating any movement toward rolling them out in other circumstances.
This might be an argument for him to console himself (or just keep smiling) in the event that he received nothing. It isn't a valid consideration for Rockstar, which should take heed of the calculus you stated with your first hand only, because for the beneficiary of a windfall, karma awaits.
Let us spin it differently: contact Rockstar before publishing the details of the fix, and offer to help them on this issue. How much do you believe that Rockstar would value the fix *a priori*?
The truth is this kind of fix is worth a lot, but it is always tempting for the company to be penny-pinching (and come up with justifications for being so) after the fact.
Its probably because some project manager would have told the engineer who wanted to fix this that its not important.
The original post, analysis and the actual fix of this bug is beyond impressive. Strangely, Rockstar awarding $10k for that is like someone searching for lost treasure, identifies historical references which leads to finding a ton of it and somehow leaves with 1/8th of it, even though they did 98% of all the work.
For that detailed effort, analysis and with a fix, I'd say that is worth around $80k. At least Rockstar was generous enough to award them in the first place; so I'll give them that.
No good deed goes unpunished. Rockstar went out of their way to thank this person and send them a check equivalent to several weeks of median developer salary. Can we not shame them for this goodwill gesture?
$80K would be on the order of half of the annual salary of most developers outside of SFBA.
I can't even get close to my current salary in my current unrelated field ($85k) to even consider a career switch. Are the jumps from junior to mid-level/pm that high? I am a PM in construction so that may end up being my path.
Compensation in the Midwest is a joke. If my family weren’t here, I’d return to one of the coasts.
Lol, I'm a lead C++ programmer in the UK and I don't make that much. It's American salaries as a whole that are completely stratospheric.
Just to give an extreme example: if Jeff Bezos uses half his income this year to build the largest estate the world has ever seen, he's still sitting pretty on the rest of it and has the largest estate the world has ever seen.
At a 35% tax rate, your take home income is $156K. At $6K per month ($72K/yr), you have $84K left over after rent. If this is you then YOU ARE WEALTHY. There is simply no getting around this. You are making more alone, after taxes and after rent than my partner and I have ever made together total in a year in our lives, and we live in a city with a relatively expensive rental market and are not "poor".
The absurd talk about "only" making over two hundred thousand dollars a year (plus whatever your partner makes if you have one) needs to stop. It's so out of touch with the norm in virtually every city in the world that it would be laughable if it weren't kind of sad.
People taking in $200k are not in any sense "very, very wealthy", except from a third-world viewpoint. $200k in today's dollars approaches what would have been called "middle class" a few decades ago, when there was one. Now you are only low, below $300k, and high, above $1M, with very few left between.
(In countries with regular health care and education, life costs less, so this about the US.)
The plutocrats have been very successful at deceiving American voters that they are not almost all now lower-class. The Powell Memorandum, 50 years ago, laid out the plan. It has been followed since then, and Americans now are routinely persuaded to vote against their own interests.
Wealth produced in the US has been rising as fast since 1975 as it did between 1945 and 1975, but normal Americans' income since 1975 has been flat. The whole difference has gone straight into the pockets of the extreme rich, and they are who are very, very rich. You and I have no concept of how different their lives are.
In 1975, regular parents could pay for putt their kids through college while paying their mortgage.
Mate, I don't know what to say. You live in a bubble made out of solid stone granite. Literally anywhere else outside of US that kind of salary makes you "very very wealthy". Within top 1% definitely. So I don't know if you think anything outside of US is third world? Or what?
>>(In countries with regular health care and education, life costs less, so this about the US.)
So how does this work with your statement above?
In the US, it would be lower middle class, if there still were a middle class. So, instead, it is high in the lower class. If you consider it wealthy, then congratulations, the propaganda is strong with you.
I'm living in NYC right now and paying less than half that in monthly rent.
Real estate and property tax is rediculous, state income tax is high, and restaurants around here charge $18 for a hamburger these days. We're going to have to move once the kid(s) are old enough to be hindered by the bad schools. It's enough to perhaps make the high salaries not quite so competitive, and especially so if you need to pay for private school.
Salary is only a partial picture of compensation in the bay area, though. If you're a software engineer at a FAANG type company, you're potentially getting 1-2x your salary between bonuses and equity. If you joined early on at a start-up that's doing well, you're perhaps getting set up to spend your 40's mowing lawns like Forrest Gump if you want. It's enough that, even if you take very conservative career choices, you could work hard out here for a decade while living comfortably, then move somewhere else, buy a nice house outright, and work for fun while not stressing out about money ever again.
The extra time can now be spent playing the game instead of waiting, increasing revenue.
So relatively speaking, it's a low payout for a huge company like Rockstar to give on their game grossing billions.
Know your worth! Demand more! Capture the value you produce!
- Chris in Duxbury, MA
PS For context: I've been building web-related things for a living since 1998, and have always been paid fairly for it. Because I insist.
PPS Ofc YMMV... circumstances, personality, ambition, etc etc. But there is so much more opportunity to create and capture real wealth than you might imagine.
In Norway, country of «holy crap, I’m an employer lobbyist and our salaries are so high we can’t compete in the export markets», you’ll only be around this level if you’re a freelance contractor billing 100% and doing all the sales work yourself.
There are some CTO and probably a few lead roles paying this much, but for an IC you’ll top out around $120k. Prove me wrong if other Norwegians have a better view :)
This will give your company a unique competitive advantage in capturing the best talent your market has to offer (and it likely has plenty of world-class talent to offer, but many of them end up getting brain drained away to those hotter markets today). It will also allow you to stay small for longer, dramatically reducing communication/coordination overhead and delaying the need to introduce management hierarchies, which is actually a substantial competitive advantage especially for early stage companies where agility and flexibility is the key to success. If all that's not enough, it will also do wonders for retention since it's basically a semi-permanent golden handcuff.
Over the long term, other companies in your market will end up having to compete with _you_ for talent, which raises the boat for everybody and helps your local market flourish with engineering talent and reverse the outward brain drain trend.
The unwillingness of employers in other markets to compete for talent outside their local maxima, optimizing for short-term self-interest over the long term health of the local talent market that they depend on for global competitiveness, is a true tragedy of the commons situation, and is a huge part of what makes Bay Area startups so dominant.
It's also what forced my hand into moving to SF myself. It would have been extremely financially irresponsible of me to stay. By leaving 5 years ago, I tripled my salary compared to competing local offers, and one job change later I'm now making more than double of that (at a startup, not at one of the big tech corps).
Nowhere in the world outside SF/NYC is there so much VC money that can support paying developers those salaries.
I would say over 90% of tech companies make their money from local markets. You can pay your developers as much as you make from your local market.
As for hiring. You have no idea how well someone is going to work out. Some people are really good at interviews but turn out to be terrible employees and vice versa. This is why the hiring process in tech tends to be pretty convoluted and broken in general. No one can figure this out. The laws regarding hiring are also much less liberal outside of CA. Hiring and firing is severely restricted in many places.
Very few companies are in the position of being "unwilling" to pay developers more. Some with plenty of cash are even simply limited by the tax laws how much they can pay. The cost of paying someone too much over local market rate incurs obscene tax costs.
Getting to that point is a bit of a chicken-and-egg problem, though. And firing policies is a political barrier that's hard to avoid. I suppose you could keep everyone on as contractors, paying them standard rates but going to them directly, not through an agency. That would land the salaries in the region suggested, and you'd still have the ability to fire without cause.
This is fascinating to me, do you have any examples of jurisdictions where this is the case?
Here in Germany the “taxes” (they’re technically mandatory insurances) than an employer will have to pay on top of the gross salary all have a salary cap, the largest one being around €85k. After that any increase in gross pay has no additional cost. Of course my personal tax rate might take a hike but that’s not the concern of my employer.
A lot of the conversation is about the value of the developers work being high enough to warrant higher salaries. With that argument we should look at gross salaries + taxes/fees paid by the employer on top of the gross salary.
Your personal tax rate is between you and the state.
This could mean hiring 1 really amazing engineer instead of 2 or even 3 average ones depending on the realities of your local market, and pay them based on what it would have cost you to pay the 2 or 3 engineers combined.
This is always a choice you have available, unless you're bootstrapping and can't even afford to make the first hire at those rates, at which point you're probably not looking to compete on the global stage to begin with, so this suggestion doesn't really apply.
Of course, the employee then has to pay VAT (19%) on everything they spend those 55% on.
Going off on a tangent: I personally won't leave for California/SF since I have family here and won't put the obviously massive financial advantage ahead of that.
But I did consider this when I was in my twenties. It wasn't a super easy choice. Interesting to hear that someone have taken the leap. Not multiplying one's salary by 3-6x could indeed be described as quite financially irresponsible, and the alternative has to be pretty important to pass on that.
They are in the UK with a somewhat decent healthcare system, somewhat sane rents (notable exception: London) and a somewhat sane pension system.
US salaries are so high because US salaries have to cover for all three points additionally.
Examples:
I lived in the UK with all those things you describe, but I still made a point of billing my Bay Area rate because that is what the work product of a software engineer is worth on the market.
So, no. Even if you live in your parents basement and they cook your meals and take are of your other expenses, you should still approach your work life as though you were an adult.
"and you're not making over $150k/yr, you're almost certainly underpaid."
Those salaries don't exist in the UK, no matter your position or the company you work for. I know senior programmers working for Faang companies in London and they don't make this much. Like, in general you won't see a position offering above £100k/year. I know someone with 25+ years experience in the industry and they are on an (incredible for the UK) £90k/year.
The only way to reach the equivalent of $150k/year in the UK is to work as a contractor and be good enough to justify billing that much. As a normal employee you won't see that.
My point is - it's impossible to judge everyone in the world by the American pay standard. Would I make more money in the US? Maybe. Would my standard of living be better? Also maybe, but I lean towards no with this one. But it's almost certainly not true that by making less than $150k/year you're underpaid in a place like the UK.
You'll see that mid-level (L4/E4) engineers at Google & Facebook easily break 150k; senior engineers (L5) start at ~225k and hit 300k. That's about 30-40% less than what Google & Facebook pay engineers in SV.
I also saw multiple other companies where engineers with less than a decade of experience were approaching or breaking 150k - Transferwise, Improbable, Palantir, Morgan Stanley, Booking.com, Cloudflare, Bloomberg, Yelp, Babylon Health, Goldman Sachs, Stripe, etc.
Either you're underpaid, or C++ has been commoditized to the same point as PHP. If you value money over language, I'd switch. But given quite a few in my network are Java developers, I doubt language is the issue here.
If you moved into in-demand fields, service delivery or consulting, you'd see 100k GBP as normal for seniors. The fields only require mastering the tools or domain and the ability to work with business stakeholders. That is from my experience, the majority of the software world.
I don't know what your friends told you, but either you misunderstood them, are talking solely about base (and see my other comment for why >100k base also exists in those roles), or were given inaccurate information. 100k GBP would be 45-60% of the credible range for senior engineer compensation at Google/Facebook in the UK (and maybe 55-70% for Amazon). That's so far outside of the standard compensation bands that other explanations are more likely.
I worked for a US/UK company with its engineering HQ in London until recently and was earning drastically more than that. Now, I was an early stage employee and doing unique work for the firm. That position wasn't advertised - I created it. But some of the more senior engineers on my team (in the UK) were earning about $150k/yr GBP equivalent, a bit more in some cases.
Are salaries in the UK lower than in the USA, yes, absolutely. They are not that much lower than non-SF jobs in the USA.
I have two guesses - either you were a contractor, or worked for a bank where I've heard it's possible to cross £100k salary. Or third option, you mean <£100k of salary + bonuses?
The one I worked for wanted me to move to London, and I said fine, as long as you give me a comparable standard of living.
Nope, they took my Irish salary and converted it to GBP, and that was it (this was post Brexit, btw).
Obviously, I didn't move. And working with any engineers in the London office was a massive pain as anyone without family commitments worked in London for a year, and then transferred to the US for a massive salary bump.
The core issue is that FAANG don't benchmark against finance in London, which means that pay is pretty out of wack.
For instance, I got a 150K offer from an investment bank in London around that time, and I'm definitely not as useful in finance as I would be in tech.
It just blows my mind that they haven't fixed this to be honest (I suspect that they'll have to, now that Brexit has actually happened).
But even putting that aside, most senior (E5) Facebook engineers in the UK make 125-160k base salary in the UK. Even if that's denominated in USD, that still leaves a significant % breaking 100k GBP base.
Source: https://www.levels.fyi/company/Facebook/salaries/Software-En... (filter for "United Kingdom")
Normal software engineers make that in Sydney and Melbourne. You need to negotiate a higher salary or switch jobs.
If the author of the bug fix is indeed from Latvia (.lv), the median annual income of a senior software engineer in the Baltic states is currently around 30K-40K after taxes.
Wow, I need a raise.
Better than nothing I guess, but suggesting that it's OK to exploit people because of where they live is kinda crass. This is an industry where you are (should be) valued based on what you can create, not where you live.
To flip this around, if this guy lived in Haiti would you say it would have been fine to give him $1k because welp, cost of living is even lower there? Insulting.
And again they had years and years they could have fixed this themselves, at in-house prices if they wanted to. They had to be shamed into it by a third-party developer pointing out their "could be fixed in a day by a single developer" tier mistakes. O(n^2) algorithms causing minutes-long load times, repeatedly reported by thousands of customers across the game's 10+ year lifespan.
It's like giving a waiter a 1% tip. Is it nice that you gave them a tip at all? I guess. Technically nobody asked them to get you your food. Is it a reasonable fraction of the value delivered by customary standards? Not really. And in this case it's like a 0.01% tip.
Disclaimer: I don't live in a big city like Tokyo.
Even traditionally mod friendly companies will tend to sweep in and either block any derivative work that's monetizing their improvements or else negotiate with the creators to take the lion's share and republish the content under their company label. There are a bunch of examples of this but Counter Strike is a pretty famous one.
That all said, it's pretty hard to argue that making minor alterations to a product should allow me to resell it as a novel work. Fan Fiction exists - but it doesn't (legally) exist as taking a novel and taking on a few new chapters then reselling it without giving a portion to the original author.
Do you think the fix is actually worth 80k in terms of additional revenue for Rockstar? I wouldn't be so sure. If not though then no it's not worth $80k.
"I'm a developer and I'm paid a lot to do my job, therefore everything I do must be very valuable" is not the right line of reasoning to use here.
The only relevant fact to justify $80K is -- would slow loading times have resulted in at least $80K of lost sales going forwards? Or at $50 a copy, then starting today would there be 1,600 people who would buy GTA V, but won't because they know loading times were slow? Maybe because a friend or reviewer mentions that as a con? (And you can also add lost DLC revenue from people who gave up playing earlier than they would have otherwise.)
Honestly, that's an incredibly hard question to answer. But it is in no way obvious that it is "absolutely worth that much and more".
In contrast, $10K is a nice round number that demonstrates gratitude for their charitable work. And keep in mind this was probably a weekend's worth of investigation or so. Not many months of dedicated work, which is what $80K buys on the job market.
Diminishing the notoriously bad load times of GTA:Online by 70% could probably do that. I actually never got into the multiplayer because at the time my HDD was taking several minutes to load the game.
It doesn't matter how much money GTA V made in the past. This applies only to future profit that would be have been lost going forwards.
You are forgetting the game which Rockstar made. A loading times fix is not 98% of the final product.
One should not hold people responsible for acting "non-perfectly". Any behavior exceeding "average" should be met with admiration and gratitude, average behavior with "respect and appreciation", and anything somewhat sub-average with "mild disappointment but no surprise and some understanding".
Having a company spend money they are technically obligated to allocate to share holders, to a developer who, without any pre-existing contract or even verbal or commonlaw understanding, helped them, is way beyond "average response" for modern businesses.
Please, don't hate on the vanishingly small number of counter-examples blazing a trail in this space.
I don't think this line of thinking is helpful. Should Amazon increase the price of their VMs if you made more money with them? Should a doctor charge a lawyer a higher price for a flu shot because their time is worth more?
Of course not. Because in all these cases, the clients would just go buy someone else's service.
Of course it is helpful to consider the value of the work you produce, and you can use that in negotiations. But you are not entitled to a higher rate just because your client is big or successful.
If the developer had offered a two day consultation for $80.000 to Rockstar they would most likely have declined and said, sorry, for that money we can hire someone else to do a lot more performance improvements.
If you want to get paid based on value created, you need to negotiate. But you don't have a moral right to get more money for the same effort just because your client is rich.
that's insulting tbh
Most people don't: <https://mkorostoff.github.io/1-pixel-wealth/>
After reading his account of finding this, how long do you think it took him? 10 hours? 20? 40? That's $250 to $1000/hour for something he did out of curiosity.
I don't think anyone here can get anything close to those numbers you specify... but you could try to hire him and make a nice offer :)
In normal people's terms, if somebody did hours of labor that gained you >$10k, would you give them $10? That would be insulting, yet that's what Rockstar did.
Still I’d give him more for the significance and PR. But these things tend to be fixed rather than subjective for a reason.
Frankly for an introvert with no public online presence or audience that’s just not very appealing. There’s easier ways for software developers to make $10k: just wait for two paychecks.
This can result in severe consequences for the developers who worked on this particular part of the codebase.
I'm quite surprised they are even going to patch the game because the level of "we dont give a shit" I thought they had about this issue certainly would have prevented them from doing anything now.
A game video company employees artists, sound effect engineers, and etc. They aren't full of programmers combing over the source code for quadratic footguns.
The issue is that slow loading times was a contentious issue with many, many customer complaints (IMHO it was the #1 complaint) and during many years not once did they spend even trivial effort on trying to look into that complaint, and that's shameful. As the tostercx's article shows, if they had ever assigned any engineer a couple hours to look into why loading is slow with a profiler, that issue would have been easily found; the fix as described was a bit more tricky because the author did not have access to the source code, but within a company that would have literally be done within an afternoon by a single non-exceptional person. But they never cared enough to have someone look into it.
The thing is that this loading time issue probably didn't exist at release as it was part of something that got expanded on over time and only got bad worse by the years.
Assuming somewhat linear growth of the data the loading time would have been at least 4x shorter only half the games lifetime ago (3 years) and 16x shorter 1.5 years after release. After that period they would probably not consider there to be issues in those places in the code so nobody really looked or realized anything was amiss there.
What are you talking about? Nobody knew about it? Is that a joke?
This long loading issue was posted EVERYWHERE and was the reason so many people gave up on GTA v. It’s simply impossible that they didn’t know the issue existed.
That they chose to ignore it is the reality.
But I still think people are demonizing too much Rockstar and are trivializing the bug when it's far from obvious. In my opinion, there is a lot of hyperbole statements about this issue when it's more useful to think if your organization could or would catch bugs like this.
I don't work at Rockstar so it's true I have no idea how this bug was handled.
I just have a hard time believing they didn't know about the bug, and I also have a hard time believing they didn't understand that it was one of the chief complaints from their end users (a simple check of any forum or discussion group is all thats needed).
So, I an only assume short of an actual explanation, that they intentionally sidelined a fix for this bug. Why, is anyone's guess. If you had to ask me, management put all devs onto money making tasks like new items and store support, and fixing quality of life bugs even ones as egregious as this one, was a low priority.
Imagine now that 160 Million people use my codebase and have been suffering from this for 7 years.
The problem is that it’s existed for 6 years, affected anyone and everyone, and would have been caught and resolved by basically anyone who attempted to look into the issue with fairly low effort/experience-needed. That is, the problem is that no one bothered to look into it, despite good reason to do so.
I don't have the numbers but would have imagined that still takes the cake.
1: Some dev gets tasked with implementing the system, at the time there probably wasn't that many ingame purchases so that 10mb json maybe was a 500k file tops? So going by the O(N2) factor it would've buzzed by in a second (probably faster since being smaller there would've been more cache hits giving another factor or 2). Being at about a second or less they don't even consider optimizing it since it's not noticed among all other assets being loaded.
2: The content/ops teams adds more IAP over the years (it's a datafile so that devs don't have to implement everything), since they're adding big models,etc they don't notice too big changes and probably just chalk it up to bigger models,textures,etc.
3: Engine devs (those most often tasked or obsessive about optimizations) might be running a cut-down server for most testing (not noticing the huge loading time) and probably presume it's other assets if testing in a full env.
So in reality it's probably just an organizational oversight rather than "bad developers", now considering this public "shaming" they received they might actually place some people in an production optimization role for a while and optimize other parts if it's possible (the question is.. will they find it worthwhile to keep up in a longer period?).
“I want you to improve loading times whatever it takes”
Sometimes you’ve got to go hunting instead of fishing.
Edit: an anecdote from the professor of my college's startup program (a hundred-millionaire former CEO/founder)- "never say sorry."
I think the real advice is “don’t say sorry unless you actually mean it.” People see right through that, because your actions and follow through won’t match your words. But people seem to appreciate sincere apologies that come with substantive corrective action.
Yeah, it's always a tradeoff.
I worked at a non-tech fortune 500 for a few years and was able to see the inside of repair/warranty/disputes. These were very high ticket items and the brand was/is very highly regarded for after sales support.
In the employee handbook for repair support staff (the people communicating with clients about the status of their repair/claim) there was a passage similar to the below:
"Apologies MUST NOT acknowledge anything other than the customer's explicit feelings without written acknowledgment of fault from $MANAGER".
I asked a member of that team to decipher WTF that meant, and they showed me an example email. They were only allowed to apologise in the below way:
"Hi bob, we're sorry you feel our product is faulty".
I always found these weasel apologies infuriating in my personal life.
It’s still awful though. I would hesitate to do business with a company that doesn’t take any responsibility for their own products.
That's more offensive and insulting and time-wasting than saying nothing.
Not apologised at all comes off better than a backhanded apology IMO.
But the below isn't correct:
> Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them."
I probably didn't word it perfectly in my original comment, but the company did acknowledge mistakes quite often after assessments of the claimed faulty/broken/whatever product (roughly 30% of faulty claims were approved and either repaired or reimbursed from memory). The rejected claims were often taken to court (like 15%) due to the value of products.
I was mostly commenting on the deception of only allowing pseudo-apologies.
Well, this HN hate fest of a discussion certainly bears out that saying! The internet is a terrible place. You will get dragged mercilessly no matter what you do. The incentives are all towards sweeping mistakes under the rug.
Which is BS tbh. Everyone makes mistakes.
I think with politicians and political appointees, it's a little different because that's very much a public trust. Nobody deserves to hold political office, they should instead be deserving of it by being ethical and having integrity.
Amen. And avoiding a culture of blame (taking inspiration from the NTSB) makes it much more straightforward to put systemic measures in place which reduce the impact of those mistakes. People still make mistakes at the same rate, but more mistakes get trapped and more negative outcomes can be avoided.
The vitriol of this discussion shows just how difficult it is to establish such a culture. People really want to see mistakes as evidence of moral inferiority.
And I guarantee anyone who's written code long enough has written plenty of footguns of their own.
I'd say that by talking about mistakes without blame, I think we actually avoid people making mistakes because folks can learn from other's mistakes, and see them as something they could just as easily do, and so internalize the lesson.
This bug has existed for years and has lead to hundreds, if not thousands of people quitting the game citing this exact reason yet Rockstar did nothing.
I think "shameful" is really more appropriate. It's not like they simply didn't know this existed. I think they have known all along it existed and didn't give 2 shits to fix it.
The game probably has tens of millions of lines of code.
I'm glad they shipped. If they can enhance the startup time, great, bonus for them. Additional credit for thanking the third party that uncovered the perf issue.
I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester.
[0] https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
Not to take away from the feat and initiative of doing so without access to source code.
There are a lot of excuses for why this wasn’t fixed but no great reasons.
Bugs happen. Serious bugs happen. They will always happen. Shaming people for bugs undermines institutional robustness against bugs because it incentivizes people to hide mistakes. Furthermore, companies with blame-n-shame culture typically fault users when those users make inevitable "mistakes" statistically guaranteed by bad UX design.
In contrast, transparency is good, acknowledgment is good. Rockstar acknowledged fault and rewarded the reporter, which is difficult. That's a positive institutional arc I'd like to encourage, at Rockstar and in the wider industry.
If you stop at "embarrassing", I'm with you. But when you proceed to "shameful", that's quite different and we part ways.
Not fixing a serious, visible, known bug for six years is what’s shameful. There is nothing positive about neglect like that. It’s kind of admirable that they actually rewarded the guy but it shouldn’t have even got to that point.
Totally agree you should not shame for bugs. The shame is for ignoring this bug until one of your customers finally had enough and fixed it by patching the binary. If it was my company I would feel ashamed.
Managerial problems that allow high-impact, low-fix-time, high-prevalence bugs to exist for 7 years are shameful and are in fact poison to an organization themselves.
Don't mistake the shaming for really being about the bug itself here. The problem isn't the bug, it's the organizational culture that allowed such a massive, easily fixed, widely encountered bug to persist for 7 years in a game that has made them $6 billion dollars over the last few years.
Yes, they should be ashamed of this.
> I think "shameful" is really more appropriate.
When you make an inadvertent mistake in the future, karma is coming for you: people will speculate that you "didn't give 2 shits".
This bug existed for a very long time and was the #1 outcry from gamers. It is well documented all over the internet.
Every company I’ve ever worked at and every code base I’ve ever worked on would have made this a #1 priority. If it were a technical limitation of the devs I would agree it is “embarrassing” but this was not outside of their ability.
I simply don’t blame the devs, I blame management. Management should have allocated the resources to fix this truly game breaking bug but they didn’t because money was still flowing. Is it any coincidence that ads played during the 6+ minutes you would wait to get into a game?
> This one escaped, because stuff happens.
No, they mostly just don't listen to their players when compared with other game companies.
And if they didn't care at all given what you say then why would they care to implement a fix for it all these years later?
Sure, ideally there would be someone internal advocating for a fix like this on behalf of customers. But Shameful isn't the right word. It's a miss like any other.
I personally feel like the magnitude of this -- especially compared to how simple the solution is -- makes it a miss that's really much more extreme than most others.
EDIT: Though I agree that "shameful" isn't the right word to use. Rockstar wronged themselves much more so than any other party.
If the fix had been some super application specific super obscure thing, we would all understand.
Instead... the bug is easily describable to anyone who knows what JSON is and the fix is “don’t parse JSON the slow way”.
So I think some of the leniency afforded to the devs may be misplaced. Then again, hindsight and all that.
Would we?
Even if I don't understand what it's doing, devs doing nothing about an O(n^2) algorithm that takes multiple minutes every load is clearly a big mistake.
Even if it's something that has to be that slow, cache the result.
I’m sure GTA has unlimited bugs to look into, but a bug that turns people away before they even start should have probably been given higher priority.
That said, they're all things like "refactor the frobnicator code so it's more obvious what it does" and "make templating system log changes to server". Nothing on my list says "Look in to that major performance issue that literally thousands of customers complain about every single day". That stuff gets sorted immediately.
You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.
This example will be ammunition in internal debates about how much to weigh product pressure vs engineering suggestions that improvement is technically possible; the next time an eng says "let's look at this slowness one more time, it may be fixable more easily than you think", they might be listened to.
lol without context you'd think something tragic actually happened here.
I can't believe you're missing this huge point.
On the other hand, a few days of doing the absolute bare minimum of quality control doesn't undo a decade of actively hostile practices. They're still very very deeply in the red when it comes to their relations with the playerbase and for good reason.
Paying $10k to the person who documented this bug is a good start, but you need to do a whole lot of work to make up for the impressions caused by that kind of prolonged systemic negligence.
Have you heard the one about the country with the death penalty for all crime? It just leads criminals to maximally try not to get caught, without regard to damage done [i.e. removing all witnesses to even the smallest crimes]. This isn't what you want; even within "negative" relationships both sides lose when you don't have a gradient of responses.
A criminal prosecuted for murder and petty theft simultaneously can't really go around complaining that they got 25 years in gaol for stealing an apple, but if everyone had already known about the murder and dealt with that accordingly it'd be pretty harsh.
(not that fixing a bug is exactly a criminal act, but the general principle remains)
Since switching the file format is probably some work, maybe involving many differnt teams and network protocols, the bug was closed because it too much efford.
And then, whenever somebody complained, the old bug was referred which unfortunately never got a proper analysis.
Sad, but I saw this happening many times in the past in some projects. Since the old and wrong analysis will be in many powerpoint slides and heads, it is incredible hard to argue against or get even time to work on such a problem. Usually, you need someone who just takes some time form other projects without telling anyone and ninja fixes the problem to solve it.
Just as I'm certain that this unsolicited 3rd party fix was used to settle some old scores.
Corporate politics is a blood sport.
as-if that's the worst they could do. Suing the guy, pressing for charges with CFAA or DMCA anti-circumvention. There were far worse ways this could have gone.
Only the prosecuting attorney (someone employed by the government) gets to choose when and how someone is charged with a crime. The victim has no say whatsoever in the matter, and indeed the DA can charge someone even when the victim doesn't want them to, or decline to prosecute even when the victim requests it.
Most of the time when people simply use "press charges" this is what they mean. I specifically used "press for charges" to try to avoid such pedantry. I guess I failed.
[0]: https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-...
How many people get to say something like, "Reduced GTA 5 load times by 70%"?!
Ya know on second thought, it's so unbelievable that he did this whole thing that I wouldn't be surprised if someone reading his resume thought he was bullshitting.
Might not put that one on the resume.
I don't think it's got anything to do with capitalism, we just tend to forget things quickly.
Millions of people play this game every day. This lone man, driven by his love of solving puzzles, is single handedly unlocking potentially hundreds of millions of hours of “lost time” across the globe.
[1]: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
Once the vendor fixed the issue it shaved a few minutes off the install time that had previously just been a UI glitch (processing the large backlog of progress bar update messages).
Had a similar situation with an Electron app I worked on. App’s window was getting very stuttery at some point during a long-running operation done by a third-party library, even though it happened in a background worker thread and wasn’t supposed to block the UI (which simply showed the progress of the operation).
Turned out, that third-party library called my progress notifier function (in which I used Electron’s IPC to notify the GUI) on increments that were fairly spaced out most of the time, but really frequent during some operations. Mere overhead from IPC calls and GUI updates added up to a point where everything lagged, even though the operation itself wasn’t that intensive. I wrapped my progress notifier in a throttler, which I frankly should’ve done much earlier, and that solved it.
What's the correlation between being averse to spending time on less-fun aspects of the game (to the point that you'd pay Rockstar extra money to get only the more-fun time)... and being averse to waiting for long load times, when you just want to play a quick session of more-fun?
Given Rockstar's investment in a steady stream of new big-ticket in-game items, and their huge revenue from that, I wouldn't be surprised if the slow load times cost them many millions.
If I was a developer at Rockstar and played this game I would have have spent my own time to fix this problem just so that I didn't need to sit around and wait every time I played the game.
It's why TurboTax is so darn slow and hard to use, wasting user's time.
Ironically the hardest thing I needed TT to do this year was handle my wash sales and it didn't even blink as it imported the 1099s. Wait, maybe it didn't handle them correctly... See what I mean? Now I better double check.
edit: Akchually, the DRY method of doing this would be to make a function like:
/* Remove this when it's time for a performance improvement! */
function Optimize(int N)
/* do stuff N times (so you can limit the amount of performance improvement) */
end
Cool fix, nonetheless!
Had they bothered to fix this early on as many of us begged them to do from day one of online's release, many of us would have stuck around far longer.
Oh well actually the shark card garbage coupled with ever-increasing item costs, as well as releasing the casino ~5 years late probably would have made me stop anyway.
Most companies won't do more due to competition or embarrassment.
https://www.folklore.org/StoryView.py?story=Saving_Lives.txt
(10 seconds)*(5 million)*(365) = 564.1 years
((10 seconds)*(5 million)*(365))/(human lifetime) = 7.775 lifetimesBut 72 years about sounds right for US 1983 Mac users, who were already closer to middle aged.
But all bets are off with time traveling macintosh users.
"It went from Dilles Bottom, Ohio to Moundsville, West Virginia. It spanned nine hundred and 12 feet above the Ohio River. 12,000 people used this thing a day. And it cut out 35 miles of driving each way between Wheeling and New Martinsville. That's a combined 847,000 miles of driving a day. Or 25,410,000 miles a month. And 304,920,000 miles a year. Saved. Now I completed that project in 1986, that's twenty-two years ago. So over the life of that one bridge, that's 6,708,240,000 miles that haven't had to be driven. At, what, let's say fifty miles an hour. So that's, what, 134,165,800 hours, or 559,020 days. So that one little bridge has saved the people of those communities a combined 1,531 years of their lives not wasted in a fucking car. One thousand five hundred and thirty-one years."
Glad to see Rockstar addressing this; I will try the update once it is released, and maybe I can play some online games without getting frustrated.
I mean the PC version was released one year later. Maybe by another team.
If you're interested feel free to reach out and I might be able to help you. Contact details in my profile.
glad to see I am wrong. many GTAO players will definitely be very happy once the new update is live.
Darn, I got the exact same issue ten years ago, with sscanf on Linux/glibc. The sscanf was forcing the huge XML mmaped file region to be entirely read.
The reason behind is that the glibc library was using some "smart" streams (to have generic functions for file/string streams, I guess) that basically all do strlen as first step to buildup this stream structure.
What's that?
So I don't think it relates in this tech situation. Tech-wise GTA5 is not that interesting for few years now.
How well they had it running on the 360 is absolutely amazing given the limitations but nostalgia is taking over if you think most games on 4th gen Xbox don't even come close.
World size? Lots of open world games are as big or bigger.
Graphics? I thought this too, but then I had another look (been a few years since I played it) and its quite dated looking now. Not bad, but definitely not on the same level as more recent games. Understandable though, it came out an entire console generation ago!
Even gameplay-wise, yeah, it has much better driving mechanics than most non-racing games, but shooting and such is fine, but nothing special.
Its a great game and was absolutely fantastic for when it came out. I'm not at all surprised that its as popular as it is! But "most games still don't come close" just isn't true, at least for large AAA titles.
Does it have to beat it on all points for it to be valid? I mean, since the games aren't trying to be GTA, they're not going to have the exact same priorities.
What about large, beautiful, but not open world games? God of War 2018 and Dark Souls 3 meet that criteria for me.
Or open world games that aren't as large as GTA but hit all the other points? Spider-man 2018 is a great example.
Or maybe large, graphically gorgeous, but the driving is terrible? Cyberpunk 2077. (Buggy too, but I assume that will eventually be fixed)
Those are just some games that spring to mind. I'm sure I can think of some more, especially on PC (the games I listed are all ones I played on PS4 -- so I'm comparing GTA V on PS4 against these PS4 games, perhaps GTA V on PC could be modded to visually beat these games)
The amount of free skilled labor businesses benefit from and abuse while not donating back to society is laughable. It's great they helped the GTA 5 community and a fun read but keep your pride and fame, I'd take cash anyday over a pat on the back.
From their blog post about it: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
Edit: I reread the article and I overlooked how there was a second issue of a poorly implemented "hash table" that was a linked list that checked the entire list for dups in every update. Since we don't have access to the source code, it's hard to say if this is poorly implemented piece of code on Rockstar's part or someone else. It could be some deeply buried piece of code that no one ever looked at for years until now.
https://www.youtube.com/watch?v=UsmqWSZpgJY
But this is online GTA. I can imagine that they handed it off to another department that didn't have the same amount of discipline and expertise. it's probably an off shoot team that creates content for the engine but don't necessarily need to dig down into how the engine works.