What was the technology stack driving the original Ultima Online servers?
quora.com
quora.com
"As Raph also notes, there were no databases originally involved in the storage of game state or player data for UO (disregarding analytics here), everything was kept in flat files. Backups worked by flagging a moment in time where no one was allowed to cross server-boundaries -- during that moment, each areaserver was commanded to fork(), essentially duplicating itself in memory (it's more complicated than this, thanks to Copy-on-Write, but let's simplify). After everyone had fork()ed, the "lock" preventing boundary-crossing was cleared. Then each areaserv began to dump out its huge chunk of memory-state into a file on an NFS server. Those files were then all tarred together and kept as a "backup" of the state of the server. These heavyweight backups happened at half-hour intervals, I believe."
"Server-boundary crossing edge-cases and race conditions persisted for a long time -- allowing for gold and items to be duplicated, though I think we had largely eradicated the big ones by the time I had moved on to UO2."
You can get repository level transactions by committing on each consistent state (squashing before push) this lets you do undo/redo and recover from failed operations easily - databases sort of give you this - but only for the stuff they actually contain (so unless you're putting your asset files in to the DB you don't get this).
you can get a long way with join, awk & grep
This model apparently allowed for quite a few dupe bugs, one of which I published on an old fan site. It was quite involved, in fact I wrote it up so deliberately convoluted that what you were doing was obvious. Did it work, I wasn't quite sure but I did suffer both duplication of items and loss from their system through no effort on my part.
Its kind of funny to learn how they did it this late, reminds me of mechanisms we employed with some early multiplayer door games on BBSes. Supporting multiple players for some early titles was a bear, the least of which was the lack of stability of most connections just plain bad code that bit you in the butt more often than not.
Once web content began to be personalized, that was no longer realistic. Databases became a required part of the stack, web servers ran applications instead of just serving file, we gained massive amounts of functionality, and haven't really looked back.
If all you want to do is run a small project, or a big one where performance matters and you have deadlines, you might just "do it by hand" and simply create your own pseudo-database out of flat files, memory dumps, whatever.
The approach described is what people nowadays would call memcache, but the entire architecture is different because client connections are persistent rather than resetting every page load like the web. They just built a standard game server keeping everything in memory in the regular data structures. Fork-and-persist is a nifty way of using the operating system's copy-on-write functionality to checkpoint state.
Freeshards usally went with a stop-the-world-worldsave. Which means they were infrequent and easily corrupted, leading to "timewarps".
Somewhat off topic, but one the pieces of code I'm most proud of was effectively a recursive XML scraper that resolved each leaf (each node was a web request) into a line (like a file path) of a massive text file, which was then grepped when a user searched for something. I hacked it together really quickly to solve a problem, which is solved orders of magnitude faster than commercial solutions that tried to be overly clever. I like it because while it was gross and bad and would be embarrassing to read, in reality grep is crazy fast and the OS cached the text file once you'd read it once. It's a nice personal reminder to me that you don't always need to build some Grand Design, especially when you're just building a tool that has no future evolving path.
The game's fiction included reference to shards of a wizard's crystal that created duplicate worlds -- a convenient explanation within that universe for the need to run multiple, duplicate servers with different users on them.
When I first heard the term "shard," I thought we were talking about UO. I still think about it every time time I hear "shard."
I don't ever run into anyone who has played UO either, and so everyone looks at my like I have 3 eyes when I explain the connection. haha.
In my mind no game will surpass closed beta era UO. It had the sandbox and freedom combined with q culture that was actually into the idea instead of the grieving
"The word "shard" in a database context may be derived from the 1997 MMORPG Ultima Online, which referred to its game servers and their associated databases as "shards." This was in reference to the game's fiction, in which the broken shards of the evil wizard Mondain's Gem of Immortality each contained a complete, living copy of the world.[24] This usage spread informally through the video game industry, and may have come into mainstream usage via Flickr, which began as a tool for an unpublished MMO."
Original blog post: http://www.raphkoster.com/2009/01/08/database-sharding-came-...
It's really hard to have a discussion about UO without the urge to talk about how important it was to me (I've noticed I'm not the only one).
So rather than resist that urge... Ultima Online is directly responsible for me being a programmer. I was 13 when I got a copy of the game and an account thanks to my parents. It wasn't the first game I'd played where I'd wondered how it had been made but it was the first game I'd played where I couldn't even conceive of how it could possibly work. It was so different from anything else that I'd played before, including the first 8 Ultima games. Trying to figure that out led me towards programming in general and to SphereServer, the UO server emulator. The scripting system for SphereServer allowed me to change the game (for me and my friends only, but still) and really explore what's possible. By the time I was 16 I knew what I was doing for the rest of my life, even if it wasn't necessarily anything related to games.
STARTLOOP
KEYPRESS F1 // client macro key for hiding skill
WAIT 1000
MOVERANDOM 4 // move 4 times in a random direction to get around server checks for using a skill in the same spot
WAIT 8000
ENDLOOP
This gives a good summary: http://www.uoguide.com/8x8
I remember a specific time when I was setting up my mining operation. I took the time to figure out how long it took for ore to re-spawn, and had my character mine in an elaborate circuit several times before starting the smelting process. I was super proud of myself, haha. I think I was about 12 or 13 at the time.
Creating phishing pages so that I can brag about hacking my friend's account was the other motive.
I think that MIT's scratch is a very good platform for children to learn about programming.
There was one website, Taran's Scripting For Dummies, that was pretty much my bible.
...I'm still sad about how awful the ultima (offline) series ended, though
PS A different topic altogether, but for quite a while it seemed that whatever clock speeds we gained was wasted on pushing pixels around on the screen...
Probably, as the term "shard" was part of the backstory: a gem that contained the world ("Sosaria") was shattered, each shard then contained a version of Sosaria.
They had a massive 10 Mb/s dedicated to them (with the ability to burst to 50 Mb/s) - bandwidth shaping was done with a Packeteer.
Thanks for that, I needed a nice laugh today.
Translation: "too old" | "too expensive for us"
First of all, this was 1997 +/- a year. They amount of stuff you could pull off the internet was radically smaller than today. The number of solved problems was radically smaller. So basically they went were few had gone before and created something successful that persists to this day. So heir career would be looking pretty damn good I would say.
Second, tech stack != career. I'm seeing this sentiment more and more on HN and it's weird. Especially weird in an industry know for its rapid change.
Your question is so backwards it almost feels like trolling.
Everyone(who's been around long enough) knows that the technology churns and real progress is slow, but I think this kind of fashion is based on employee marketability - nobody wants to be just a code monkey, after all.
This works splendidly because even the hippest startups working out of mansions by the beach (or whatever) are still fairly kludged together from an engineering standpoint, especially if there is ageism serving as a kind of experience cap (for example ageism by proxy of being too expensive for those startups to pay for). People are extremely territorial about their piles, but if you are good enough friends with those engineers (or get them drunk enough), you find out that everybody lacks confidence in their work to some degree. Someone I work with currently said that if you don't look at code you wrote three months ago and feel like it's shit code, you aren't improving. That is a very insightful comment, to me.
On the other hand, UO burned me on game development worse than any other game I ever worked on (though I have kept working on games since then). It was a miserable project. So the effect on the resume was good; the effect on the soul was awful. I never worked on another MMO again and never, ever would.
Making games is not as much fun as playing them.
Do you want to know about...
1) the incredible complexity of MMO development and why nobody wants to make an MMO (though everybody thinks they do)? 2) the origin of Ultima Online as a small, skunkworks project that led to extremely awkward core engineering? 3) the even more awkward munging together of the Ultima IX and Ultima Online teams that led to a suddenly massive team trying to ship a massive game in six months (and succeeding!)?
Pick one.
Ultima Online had begun as a small band of junior developers within Origin back in like 1995. The original concept was to build a massively multiplayer game using a variant of the Ultima 6 engine. That was a pretty far throw back considering that Ultima 7 and 8 had shipped with much more advanced features and Ultima 9—fully 3D and "modern"—was on its way. The group of programmers who worked on Ultima Online for the first few years were brilliant, talented, and driven, and they produced a lot of great work. The game would, of course, never have happened without them. But, they were absolutely green programmers, fresh out of high school in some cases, and... well, let's just say that the Ultima Online engine began to sag under the weight of its massiveness by the end of 1996.
One example just to illustrate the idea. In most game engines even at that time there was a conceptual difference between The Camera and The Player. Though the camera would often be attached to the player, you could normally move the camera someplace else—to show an in-game cutscene in which the player wasn't involved, say. But in the Ultima Online engine this distinction was lost. The camera was the player: where the player was, that's what you were looking at. It meant that when we wanted to show a cutscene, or a clairvoyance effect or suchlike, we had to use a hack: make the player invisible and non-colliding, teleport the player to the scene, move it around like a camera, then teleport it back to its original position and reset its usual state.
This sort of thing was commonplace. Imagine this times 100,000.
So here we have several young, gifted, powerful programmers making history—as they know, though hardly anyone else did—building a brilliant but difficult concept into a working implementation, often naively, sometimes shoddily, but effectively, productively.
Late in 1996 Electronic Arts (who owned Origin) woke up one day and realized that they had a massive potential hit on their hands. A game they could sell not just once, but once per player per month. Game subscriptions were a whole new concept, and boy what a tasty one.
After waking, the next thing EA did was to demand that the game ship by summer 1997. That gave Origin about six months to ship a huge, complex, wildly innovative concept that was barely out of the prototyping phase. This was, to say the least, an ambitious goal.
So they threw bodies at it. The Ultima IX team had been working on "the sequel of sequels" for quite a few years. The team was large but progress was slow. Although Richard Garriott would never let the game die, EA had no real enthusiasm for it (and indeed in the end it shipped, and was quite good fun, but buggy and didn't sell well).
EA put Ultima IX on hold and moved all the developers onto the UO team. That more or less quadrupled the size of the UO team overnight.
So now you have a small, brilliant, fast, naive cadre of programmers for whom UO has been their baby for a few years squished together with a large, perhaps less brilliant, slower, but wiser and more experienced pile of developers who are pissed about losing the project they've waited their whole to complete and find the idea of shipping any game, much less the world's first bona fide MMO, in six months positively laughable.
There was fear. There was anger. There were frenzies of productivity. There were frenzies of laziness. There was hope. There was dread. There was culture shock. There were absolutely awful tools. There were absolutely indecipherable codebases spanning Windows and Unix, with no documentation and the only people who understood it were too busy (or just didn't care) to answer questions. There was a game that could never live, could never run, could never be tested or debugged on a single system—a game that was intrinsically an MMO and needed a shard to run.
There were tiny bugs that took many, many, many weeks to find and fix.
This feels like the right time to quote Dickens: "It was the best of times, it was the worst of times." Only it wasn't. It was just the worst of times. And despite all the joy that UO seems to have given other people, it was a loathsome development experience—I can't think of a worse or more accurate word than loathsome—and after it shipped in September (was it?) of 1997, just about on time, miraculously quickly, I never played it again.
Hell I'd hire any of the guys now if that was in their resume somewhere, because that was some nifty stuff. Gotta respect those sort of individuals who show real front-to-back understanding of what it takes to build/implement something people actually use, do it well, and adapt to change while scaling.
I remember it being very laggy and a bit glitchy, despite having a high-end PC (50mhz 80486, 28.8k modem). But I was still mesmerized with the concept of an MMO, that I could see and talk to and fight with people from anywhere in the world.
(Remember, this was 15-20 years of Moore's law ago so price/performance was about a thousand times worse than today)
that's so awesome. I love little stories like this from early pioneers.