Diablo III Economy Broken by an Integer Overflow Bug
minimaxir.com
minimaxir.com
Now, in Asheron's Call, the world is huge. There are hundreds, if not thousands of vendors. And three of these vendors were set to sell their goods in stacks of up to 1000. Unfortunately, Cost of Pyreal Scrab * 1000 > 2^31, which wrapped. I can't remember if you either just got the goods for free (which you then sold back for huge profit), or if you actually got paid to take these things. Either way, overnight, the economy was destroyed. The entire game state had to be reset from backups; a dreaded rollback. Worse, the developer took a few days to do this.
Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players. Even though I was just a player, I can still remember the outrage all these years later. It taught me to always use appropriate types for objects with "value", and I've never accidentally used signed or floating point storage for currency again.
A nightmare scenario for the developers in both cases.
Welp, that's what happens when you have only online play on only official servers. Single player offline wouldn't be affected - cheat all you want! Online play on unofficial servers means server admins can take whatever action they want - ban offenders, leave offenders alone, or rollback - depending on what the admin and the players want.
Consider this completely anecdotal, but I think that around the time this bug actually occurred, Asheron's Call and Everquest were the only two 3D MMOs that were worth mentioning. I recall the delay in rollback having something to do with Microsoft bureaucracy at the time as well -- Turbine was plagued with MS as a publisher having some sort of veto power over their business and was frequently met with resistance.
(Full disclosure: I love Turbine unconditionally for creating such memorable adventuring experiences with Asheron's Call 1 and 2)
"Oops." --Devilmouse
When you say int, you usually want an actual integer, not an integer with an arbitrary limit. In this day and age, having that limit there is simply premature optimization.
I think having a nice bignum type--one that looks and feels just like a normal numeric type--is very important. It should also probably be the default; you should only switch to a machine type if you have a good reason. With gmp, big integers perform well enough to be used widely.
You say this with certainty. Do you know of studies of real-world programs where machine-sized integers were replaced whole-sale with bignums?
There has been lots of fuss and even pushback about the change to use unicode instead of byte arrays, but I have never heard anyone complain about semantics OR performance of the switch to a single integer type.
I assume this was probably a server side bug, since all the accounting would never be trusted to the client side.
If you are writing highly-performant server code, the actually memory size is extremely important. You cannot (should not) abstract away the machine specifics of the datatype if you want to write optimized code.
In some cases where the underlying datatype isn't a concern (e.g. Javascript), I agree with you. But ultimately, this isn't a failure of technology, it is a failure of the software development process.
However, the physical size of the integer as stored in the CPU cache, RAM, and HDD is still going to be 2x as big for a 64-bit integer. In a hypothetical worst case, you are cutting your CPU cache and memory bandwidth in half, which is tragic.
Additionally, while most physical servers are x64, the OS, server software, and virtualization layer is still often 32-bit -- maybe for legacy, maybe for performance, maybe for a lot of reasons. Upgrading that whole stack up to 64-bits just for the luxury of having default 64-bit integers seems misguided.
I don't think you can just throw that out here with no story. Well...I guess you can, it just makes me sad.
It was fairly easy to track down the bug once we passed the initial disbelief stage, and I don't think many clients noticed. But it could've been a lot worse - I'd been so close to running this last-thing on a Friday before I left.
Why wouldn't currency by handled by the type system? You could still have an overrun. But it'd be handled more appropriately.
Long ago, I wrote a budgeting / estimating tool. Costs were represented with binary coded decimals (BCDs). Not floating point numbers. Just like an accounting system.
Competing products could have weird roundoff errors. Not mine.
That file was a customer log that had grown to 100+gb in size due to an error that she was debugging. She failed to check the log size, instead assuming that it was a small file left over after that night's log rotation. When vi tried to load the file to memory, it almost crashed the box before we could kill it (we still got calls about degraded performance though).
http://kol.coldfront.net/thekolwiki/index.php/Black_Sunday
You could then sell all your new shiny missiles for loads of money. Made a hard game a bit easier.
Another one in FE had you put in passenger holds, fill them with passengers, then sell the holds -- this would obviously not work as you had to evict the passengers first, however the game logic credited you with the cash anyway because the check came after the money had changed hands.
Most Super Mario games requires way more abilities than that and less time.
Don't waste your limited time on earth playing consumption-driven games. I've been trying Eve online for a few days, It does not looks promising, it seems that Eve also is also driven by item accumulation and not actual playing.
Then I started Guild Wars 2 a few months later, and played that to death. :)
Many of my small-scale engagements took several minutes to resolve, one way or the other.
Not to mention fleets. In fact, one of my fondest memories was taking part in Agony Unleashed's PVP classes. That took several hours of intense playtime - we had to have multiple 'bio breaks' for people to be able to go to the bathroom and eat something - while being guarded and with scouts to give early warnings to the fleet. And it was amazing. Instructors teaching game concepts and military-like tactics and command and control (and actually enforcing discipline).
I've never seen a game getting even close to that sort of organization and discipline. CCP should hire those guys as teachers to introduce players to more advanced game concepts.
I felt that they just took a ton of ideas from WoW and implanted them into Diablo. Sure it may have made them more money but it made it a lot less fun.
For this very reason I must on no account ever install it.
I didn't see the connection so clearly until now. How is D3 even legal?
I guess with all the hysteria about video game violence, no one has had time to actually look at what's going on.
ETA: I'm guessing that the fact that there's some skill involved makes it more okay. Like a fly-fishing tournament.
Also, by that logic, buying a pack of baseball cards is gambling.
It's about hoarding, alliances, corporations, mining ops, PVP ops, big alliance battles, 0-sec space mining/pvp ops (this is the best part of the game).
Next thing you know I was crawling all popular market hubs in Eve, storing price history of each item in mysql, and programmatic analyzing the data to find the best trade routes for profit.
Then I realized I need much more data, and prepared a small data-grabber client for other people to run, as well some cloud storage to upload it to.
I looked at the calendar and noticed 2 weeks have past and I didn't do much else, so I came to the conclusion this might not be the most productive thing to do and quit Eve :) Problem is, I can't play these games the "normal" way, when I see it got APIs etc. I just have to go all out on it - or just not play at all.
http://massively.joystiq.com/2010/09/17/new-eve-exploit-give...
Is this an error? :) Although I assume the players would very well enjoy game-breaking exploits, as long as they are to their advantage.
Good items sell for hundreds of millions. The number of zeroes doesn't matter, as long as the balance between items and monetary value is stable.
And you're not just starting the game. The way that D3 drops work, the first few times you're visiting the auction house, it's to buy, not sell (unless you get very lucky).
The game was also balanced around the auction house - meaning fewer high quality items drop, with the belief being that you're selling those few high quality items & kitting yourself out from the auction house.
When I first played, the difference between playing with just the drops I got versus kitting myself out from the auction house were vast. Some people might enjoy the challenge of just playing with dropped gear, but just as many do not.
AFAIK, Blizzard collects interest on each real money transaction, so if they're optimizing for maximization of that interest (I'm assuming they do) they aren't minimizing occurrence of the rarest of the items. They should behave like diamond cartel...
Did in this particular case ;)
Compare this with World of Warcraft where they deliberately built in mechanisms to keep destroying money through consumables and bind-on-equip items.
Also, AFAIK the only parties who exploited this bug were gold-farming bots. The computers won't mind, especially since their owners probably made bank off this.
Good luck overflowing a 64-bit unsigned.
It's entirely conceivable that people have billions in currency, but not quintillions. Though perhaps Zimbabwe serves as a counter-point to this. They've had to slash twelve zeroes off their currency valuation on at least one occasion.
More importantly, blindly increasing all of your 32-bit integers to 64-bit is going to double your memory usage, and ultimately just mask the real issue (i.e. improper bounds checking).
They made a mistake for sure, but 32-bit vs. 64-bit architectures should not be on trial.
The real impact on memory usage is likely the 8-byte pointers, but if you have a non-trivial amount of memory, it's rarely an issue worth fretting about.
If we cared about pointer sizes, we'd still be writing 16-bit code.
Running 64-bit would have "prevented" this bug simply by virtual of that fact that the default datatype would have been big enough to avoid overflow, but it isn't really a solution. I just find 32-bit vs. 64-bit to be inconsequential to the real mistake, which was an improper software development process.
Although back in those days, if you had that much gold, you were controlling the economy by yourself anyways.
In this case, how should they defend against an overflow? Impose an arbitrary limit on gold?
In the short scale, you'd be right, but the author probably uses the long scale.
See http://en.wikipedia.org/wiki/Long_scale#Comparison.
It's one of those things that makes English ... interesting to learn as a foreign language. :)
EDIT: Actually no, I mis-read the table on the Wikipedia page. The author was simply mistaken, and has corrected that. Thanks to everyone who pointed this out.
Source: your link.
It's things like having three different "gallons" that all mean wildly different things that make communication difficult.
It sounds so strange. Are you sure this isn't another Wikipedia hoax masterminded by Stephen Colbert?
There was another screenshot which has an account balance in the trillions that Blizzard argued was fabricated. Given the ease in which this trick was performed, I wouldn't be surprised if it was authentic.
This explains /everything/.