The challenges of Creating a Civilization Web Game | Hacker News Reader

The challenges of Creating a Civilization Web Game

code.google.com

81 pointsroschdal74 comments

My solution: drop support for IE. No, seriously. I understand that it has a very large "marketshare" (installation base) but I think that very often IE is used on office desktops, that is places people wouldn't use for playing games anyway.

While there were times when IE was a decent browser it is clearly standing in the way of innovation at the moment. It is very easy to download a modern alternate browser and a large number of websites using the canvas element would force Microsoft to build a fairly good implementation of it into IE 9 (as it was I believe hinted by the IE project team leader anyway).

By the time you finish and have it running smoothly, IE9 will be closer to completion (if not already released). Then you already have support for IE.

So unless there's a burning desire to market this to IE users right this minute, save yourself the hassle and let IE catch up with standards while you work on a better javascript client with Canvas.

> ...IE is used on office desktops, that is places people wouldn't use for playing games anyway.

Pull the other one!

My startup was based around Flash-gaming, and our traffic numbers (to the extent that we had traffic, which wasn't enough to keep going but was enough to see some patterns) showed that the vast majority of hits occurred between 9 AM - 6 PM Eastern, Mon-Fri. Tailed off until about 8 PM (my guess is those were the west-coasters), then we'd have very few hits at night or on the weekend. Funny how that works. ;-)

OK some games, but I am yet to see anyone playing Civ 2 during work hours, or Alpha Centauri. That would sure make me laugh.

And then cry, if they are on your team. 5-10 mins of a flash game during a break ? No problem, it will help them relax. Several hours of FreeCiv (and Civ is a huge time-sink) ? Not during the workday, God no.

But I promise I'll stop once I discover Gunpowder!

I agree with the suggestion to drop IE. If someone is willing to dedicate enough time to play through a Civ game, they're probably not at work and probably willing to install a third party browser, so you're not sacrificing too much potential "market share," and saving yourself some grief; assuming Canvas performance is sufficient on FF3/Safari4.

Some other suggestions:

* Investigate multipart XHR

http://blog.digg.com/?p=621

This is "bleeding edge" and may not, in the end, work, but it'd be worth looking at the work Digg has done w/ multipart XmlHttpRequests (cramming multiple objects/files into a single XHR). The demo they've posted works pretty well in non-IE browsers and it looks like they've laid the foundations for supporting COMET-like delivery of objects over a single request, and have thought about how to address client side caching of objects.

* Business model? Is this just an OSS project or an attempt to build a business (or both)? I could imagine a small business that might generate enough $ to pay some coder/admins & some freelance graphic artists to work on the open source part of the code by providing easy-to-set-up on demand game servers for multi-player FreeCiv games, perhaps w/ a sister set of web apps for collecting game stats/organizing games/etc.

Good luck!

Canvas is also too slow on FF/Linux. Actually, for much of the Canvas stuff I've done, the bottleneck has been Firefox, not IE. excanvas.js and VRML isn't that slow; it's no speed demon, but Canvas is just about as slow on FF/Win and much slower on FF/Linux.

If I were them, I'd ditch the project now. IE support is not their problem. Everything they're doing just screams "this is my first try at game development", as well as "this is my first try at DHTML", and they've bitten off more than they can chew.

As it turns out, IE's rendering engine is actually faster than Firefox's. I know this from having written a pretty heavy Canvas-based web application recently, and from a dozen years of writing video games for browsers. If he were to ditch FF support, he'd actually have a better shot of getting this thing working.

But really, he doesn't have a chance, so the point is moot.

Seriously? You're hanging out at news.ycombinator.com and claiming that he should quit developing something because he doesn't have prior experience?

Yes. Game dev is complicated. You should start off with something you can hope to actually accomplish if you want to move forward.

Building a Civ clone is not like building a Twitter clone. Especially with the terrible architecture decisions this guy has chosen. The project will most certainly fail, and he probably won't get far enough along to even learn anything.

I'd suggest he starts by building something small. Something that fits entirely in the browser, so that he can learn the basics of DHTML, Javascript and Canvas. Something small in scope so that he can learn the basics of video game programming. Something that doesn't overload his brain to the point where he's asking whether he should implement Comet to handle a turn-based game.

So no, I'm not saying he should quit programming, or even quit game development. I'm just saying he should abandon this particular project until he has at least some of the skills he'll need to pull it off.

"Something that doesn't overload his brain to the point where he's asking whether he should implement Comet to handle a turn-based game."

Freeciv is a turn based strategy game, but it uses simultaneous movement; everyone moves their units at the same time. Therefore updating the mapview must occur in real-time.

Using Comet allows the web server to push data to a browser, without the browser explicitly requesting it. So when an event occurs on the server, it will immediately be trasmitted to the clients with Comet, rather than waiting for the polling interval of Ajax + the delay of a packet being transferred from the client to the server.

So in my mind, Comet could speed up the time it takes to transmit a packet from the server to the clients.

"Especially with the terrible architecture decisions this guy has chosen."

How would you do it better?

Given that I'm not exactly experienced at game development either but I find this discussion really fascinating, I would be interested to know why his architectural decisions are so horribly wrong.

I suggest dropping the project, since civ has cost humanity enough already - browser civ could cripple my productivity... just a few keystrokes away when already procrastinating. I both love and fear this project.

I had to write a freeciv patch for my girlfriend which adds a daily move and/or time limit to the game.

It was much appreciated, still have a couple of odd behaviours that I'd have to iron out before letting it out into the wild.

Well, the first thing I would suggest is putting all of the images into one file then using CSS to control what portion appears.

That should cut down on both latency and issues with IE only downloading one image at a time.

Several hundred CSS sprites in a file is pushing it. It would probably be a better idea to keep it to a ~20 or so per file. Plus, if I remember playing Civ, you can lazy load the suckers -- when you start out all you can see is a lot of black, a settler, and maybe 6 or so unique sets of tiles, plus the game interface. You don't need to block on waiting for the battleship icon for quite a while.

Civ also has some features which make it very easy to cheat creatively regarding logic for loading those things. You can guarantee, for example, that no Riflemen will show up anywhere on the map until someone has discovered Gunpowder. Thus, if someone discoverse Gunpowder, you send everyone an <img src=".../gunpowder_units.gif" style="display: hidden;" /> and you'll have the Riflemen sprite available instantly once someone actually constructs one. (Edit: There are, of course, security issues associated with that example.)

Internet Explorer has a limitation where only one image can be downloaded at a time

Really? Are they sure they don't mean "IE only downloads two resources in parallel from any particular domain, like suggested in the HTTP spec"? This has a fairly simple solution, familiar to anyone who has read the YSlow stuff: create a few subdomains to alias your HTTP server, load images from a few of them at a time. I do it for my website, works like a treat.

I would sprite each set of tiles, based on the dimensions. So you'd have one image for all the terrain tiles, one for all the terrain decorations (grains/shields/whales/fish/game/etc.), one for all the improvements, one for city styles at each game era, one for units (again based on game era - put your phalanxes/warriors/settlers in one sprite, your battleships/airplanes/carriers in another), etc.

This also avoids some nasty binpacking problems around combining sprites of different sizes.

When the game starts up, you only need to preload the terrain, terrain decoration, and initial unit sprites - that's probably a couple dozen, no more than you'd need to draw the page as an image. Less, actually, since you can re-use your terrain sprites.

You should then be preloading the other sprites as the game goes on. If the user isn't doing anything, use that network connection to download the next spriteset the game will need.

This technique is often called "css sprites" if the author wants to read more about the technique.

So you are suggesting to stop using the HTML canvas element, and instead use Javascript to render tiles which are clipped using CSS from one large image?

I think you could do both. Position 96x48px divs with css sprites over the canvas element to draw the images. I think this would be a performance win too, because browsers are already very good at drawing things like css sprites (although I don't know how well it combines with the canvas element).

What would you end up drawing on the canvas, if the images are drawn with sprites? I can't think of anything else in Civilization that gets drawn, other than text for city names (which should probably be done as absolutely-positioned spans).

I think you can do sprites within canvas.

You can, but it's quite a bit slower. IIRC I benchmarked drawing a 32x32 image on canvas at about 15ms, while setting the top and left properties of an img tag or div with a background image was under 1 ms. The absolute numbers mean nothing (it was on an old laptop under Linux), but the ratios indicate at least an order of magnitude different in performance.

You can use sprites within canvas, but excanvas doesn't support it. Therefore it will not work in Internet Explorer.

I think that was a response to the "How to handle a graphics tileset which consists of many 96x48px tiles?" question.

A somewhat related idea: Shouldn't it be possible to implement "canvas sprites" (no CSS involved) by downloading a large image, carving it on the client side, and drawing the sprites into a canvas?

This is a cool project idea. My first impression is that this feels like a natural fit for GWT. The ImageBundle mechanism would allow you to automatically build css sprites at compile-time -- and the logic for accessing the individual tiles from the sprites: (The gwt-canvas project is a small wrapper that would allow you to draw instances of the Image class to the canvas. http://code.google.com/p/gwt-canvas/ )

It would also make it easier to move larger chunks of the FreeCiv client into the browser in a manner that would be easier to refactor and maintain than Javascript - and while being compiled to be as small as possible.

Java, for all its faults, is strong in refactorability, which would be essential for keeping the code in sync with a moving target like the current C implementation.

My first thought too - this also might be a perfect fit for something like vaadin where you do everything on the server side and push it to the client gwt for rendering.

I think the real question is, where will you possibly get sufficient cleavage to advertise it? http://www.codinghorror.com/blog/archives/001286.html

God I hate those ads. They pop up everywhere. I hope that game dies.

Can you do some kind of differential download and local move? Eg. when the user moves his mouse to the left 20 pixels and the world scrolls, then you only have to discard 20xheight pixels on the left and fetch 20xheight pixels from the server to bring in on the right and move over the entire canvas (or whatever) by 20 pixels, without calling setpixel() for each moved but unchanged pixel. Then add any other fluff such as a building exploding or whatever.

I think that'd actually be slower if it means you have to do the drawing on JavaScript (vs. the browser's native rendering engine). As a rule of thumb, anything on an interpreted JS engine (not Chrome or FF3.5) runs about 1000 times slower than the equivalent C.

The way I'd do it is create a bunch of large tiles (like the size of Google Maps) and pre-draw all the sprites onto a single tile. Put those tiles into a single container div that is bigger than its container, positioned absolutely, and clipped with overflow:hidden. When the world scrolls, simply change the positioning of the container. If it scrolls far enough that you need to render another tile, drop the one off-screen and pre-render its replacement on the other side, so if the user scrolls even more, the images are waiting for them. Much like how G!Maps does it.

This offloads the graphics work onto the C++ rendering engine, which has been tuned for this sort of stuff. Doing it in JS is just asking for trouble.

That might only work for the map though. I wouldn't want to have to download a pre-rendered tile every time that I moved a unit. It would make more sense to just pre-render the map tiles, and possibly the city tiles. Then the units would still have to be downloaded to the client and drawn on top of the map.

The same goes for changes that occur to the map (at least changes within the current view). Map tile improvements would be a pain if you had to download a section of the map just to update a single tile. It would probably make sense to have the sprites on-hand to update the currently viewed map section, and have it automatically added to future pre-rendered tile updates. This way you wouldn't have to download the map you already have to see the improvement, but when you downloaded future sections (or even come back to the current section) it will be part of the pre-rendered section.

All of this is disregarding whether or not there is a mode to show the map without improvements on it (IIRC, there was an option like this in Civ2). Then you might have to change these design decisions to accommodate that unless you're ok with the user needed to re-download all map sections whenever that options is flipped on/off.

Oh, by pre-rendering, I meant using DHTML, not images. Each tile is a div. On the div, you draw a bunch of absolutely-positioned divs (sprites) with transparent PNGs for backgrounds. The sprites themselves are layered, i.e. you'd draw a bunch of divs with background images for terrain, then you'd draw resources at a higher z-index, then you'd draw cities, then you'd draw units, etc.

The reason for pre-rendering DHTML-based tiles is because a.) you have to worry about download times for the sprite images, though hopefully you can pay this once and have them cached forever and b.) it takes a fair bit of CPU to create lots of DOM elements and move them to the correct positions. Less than blitting an image to canvas, but more than just changing the top and left of a container div.

I've put most of my comments as inline responses to other people, but here's one more:

If AJAX is too slow, do not think Comet will solve your problems. It's very hard to get Comet to work reliably on all browsers across all types of firewalls and proxy servers.

I did a prototype of something using the library that powers GTalk, Google Docs, and now Wave. Even with the library all pre-built for me, it was pretty hard to keep it from occasionally dropping connections and associated messages, and once in a while it'd lock up my browser (much like GMail and Docs do, occasionally. ;-)). I also peeked under the covers, and there's a lot of subtlety in what's going on.

Comet is best for when you have stuff that really needs to be real-time. Things like stock quotes, or chat. IIRC, everything in Civilization is done in response to user input. AJAX is fine for that.

What's the reason for not doing it in Flash?

I might have to investigate Flash further now. Firstly, at the time I started the project, there was no Flash plug-in available for my OS (64-bit Linux). Secondly, I would have preferred using only open standards for this open source project. Finally, and most imporantly, I didn't know Flash or Actionscript.

Unless the purpose is to make this into some ginormous company, I'd highly recommend excluding IE from your list of worries. If it's a love-of-the-game project, do it to your (and your friends') satisfaction.

I recently looked into using Flex to develop a Flash interface, and found it quite simple to get started... just beware of problems with asynchrony (that's what tripped me up the most).

> just beware of problems with asynchrony

Please elaborate. (I have also been experimenting with Flex.)

Many operations in Flex occur asynchronously, such as when accessing remote services. That is, they spark a remote query, and you must explicitly wait for the response (such as by binding a function to the appropriate change event) before you can continue a computation. Sometimes this occurs when you may not expect it, in which case you'll see problems such as UI controls not being updated, or exceptions from using nil values (because they are used before the response comes back).

When you do expect it, you have three options: 1) you always wait for a response before returning, making your function sychronous (thus maybe killing performance), 2) you pass the responsibility to handle this behaviour to the caller (thus creating an abstraction leak), 3) or you define a custom event that you trigger when the response occurs (best behaviour, but needs extra code).

Certainly not a huge issue once you get the hang of it, but something that can definitely trip you up when you're just learning it.

It is very similar to ajax. The problem I have is when a game gets into the 2KLOC territory and above; when you have multiple files loading at different times in the game engine in order to offer the player a better experience. This leads to many, many handlers and a lot of chaining, and your pretty UML will quickly become incredibly hard to see in the program flow.

For instance to save the user you can't just call User.Save and then go about dependent routines, because there is a possibility that call fails or takes a long time and then the player is out of sync. This is especially important if you want to try to minimize cheating, as the longer the client goes without syncing, the higher the probability for tampering, replays, etc.

It seems that Flash doesn't even have a "yield" operator. And if you try to busy-wait, the machine grinds to a halt, because Flash events do not preempt code execution.

Speaking of openness, if there was a large open source game written in Flash (such as FreeCiv), the swfdec/gnash developers might be willing to invest the effort to be able to run it.

You may also consider using Silverlight. The .Net framework on the client is more powerful and capable than Adobe offering AFAIK.

Well it might be more powerful but it has a lot quirks and little incompatibilities across browsers

Also, Java applets are due for a revival.

Ha, yeah. In a way it has already begun: the revival of Java in the browser is GWT, obfuscated Javascript is the new bytecode, and canvas is the new Java2D.

Insufficiently trendy.

If redraw speed is a problem, perhaps in-browser vector graphics (SVG, VML) would be a better solution than raster (Canvas).

There are probably a few libraries around for this sort of thing, but the one I've had experience with is Raphael: http://raphaeljs.com/

It should mean less time redrawing the map because you can just reposition the elements on the screen dynamically.

I'm going to go with no. Vector graphics are only feasible as long as the shape bytes it would take to represent an image with nodes is less than using pixels. Because many of the items in his games have gradients, and depth, I think vector art would take a lot more space than simply using pngs. Of course this is only dealing with latency; I'm not quite sure of the vector vs. raster calculation tradeoffs but i can't imagine it being very much in vector's favor with relatively few calculations per frame.

Fortunately, SVG and VML don't limit you to just vector graphics. You can render, transform and scale raster graphics within the vector graphics space. The graphic is just like a textured polygon that you can move around.

Consider how 3D engines work, where they compute all the visible surfaces using vectors, and then transform and scale raster "textures" to apply on these surfaces.

The XML issue isn't actually relevant either because normally the SVG/VML DOM objects will be generated at runtime by JavaScript, rather than serialized in the page and served up statically.

How do you describe the SVGs (i.e., the sprites) that get sent to the browser?

Here's an example of what I mean. It only works in Firefox, unfortunately, but shows the concept of using raster sprites within SVG:

http://www.mattryall.net/demo/civ-svg/

SVG is by no means a solid game development platform, but it is some interesting browser-based technology which might be useful for game developers to investigate.

Pretty smooth demo :). I see what you mean now, but i was going on the assumption that in a turn based game he wouldn't need to do much transformation, hence, my suggestion to just use pngs and allow the browser to redraw them.

Edit: also because you are using an XML format with SVG, you'll need to be conscious of parse-time (vs. binary format which is handled natively). This is non-trivial (trust me, i've tried it). Most game developers don't think like web developers: optimization is at the forefront of the development, or should be, or else you're going to have major problems with playability.

Never use a 3rd party library to do your core functionality.

This guy's whole project is going to revolve around DOM and Canvas stuff in Javascript. He needs complete control over that, so relying on somebody else's library for it will kill him.

If you're building a business app and need graphics for reporting, then sure, use a library. Use jQuery even. But if you're building a video game, you're going to need to do this stuff yourself.

The author already mentioned that he uses ExCanvas for IE. I don't think it's a stretch to use a library for what would otherwise be tedious recoding of the same logic in both SVG and VML.

If a library helps you deliver the software faster and better, use it. If it gets in your way, then consider using the layer underneath directly.

But, regardless, my main point was to consider browser-based _vector_ graphics, which are several orders of magnitude faster than canvas for activities like this.

You may want to look into how Quake Live is delivered over the web..

Quake Live uses a plugin. It doesn't "run in the browser" like one might expect something to "run in a browser."

That was part of my point. Instead of trying to do in something that was not meant for it, consider an alternative way.

This defeats the purpose of trying to create a web game.

In other words, compile the FreeCiv client for NaCl.

Unite3D delivers a kickass plugin for games.

http://unity3d.com/

But it's a commercial product and might be too expensive for the OpenCiv guys.

I don't understand why he is trying to use canvas, rather than positioning images with HTML and CSS.

I haven't coded something like that myself, but I have been wondering if CSS layers would work nicely for isometric graphics.

I am surprised how many people suggest Flash. Flash needs to die, so it is a worthy goal to try to implement games in Javascript instead. Flash simply is not available anywhere and causes a lot of suffering.

Well, with this architecture it'd be a real surprise if it worked. It's a brave attempt, but it was a long shot from the beginning.

I wonder how it would work as a java applet. Might require a jre download, but it should be many orders of magnitude faster then javascript.

Well if we're going to bring out the big guns, the plugins, this would be a different situation entirely. Besides, if a user can download a jre, why not just download the whole game in C?

I'm not sure you need a jre. If I were to guess I'd say no, but it's been awhile since I played with applets on computers who didn't have it installed.

But web games are better for a host of other reasons:

- compatibility between os-es

- compatibility between versions of the same OS (win 95 vs xp) - much greater lifetime

- no hassle with installs, play instantly and from anywhere

- easier to multiplay

- switch from "savegames" to a "profile".

I'm not sure you need a jre.

Eh? How do you run a Java applet on a system without Java (Java Runtime Environment)?

You have a default jre installed in the browser for 10 years already. The problem is it's not always the right version or flavor - Microsoft did a lot of nasty stuff in this direction. But this was a long time ago, that's why I said these days you probably don't need a separate download.

As an up to date example, I tried a yahoo card game this week-end (http://games.yahoo.com/card-games), and I couldn't launch the game in FF without installing the latest JRE plugin. The game in question if you wanna try is "pyramids".

Try these options:

1. Go with SVG and WebSockets.

2. Go with pure HTML,Javascript and CSS with transformations.

I am working right now on a game based on SVG and couldn't be happier. Works flawlessly on FX, WK & OP.

One last advice, drop support for IE and save headaches and wasted resources.

Reply on news.ycombinator.com