Hacker News Reader

4GB is really pushing the boundaries of what you should be using HTTP for. It's a nontrivial download for users with slower connection speeds, so you should really have better failure recovery than a simple HTTP download provides.

You should distribute it in pieces, or use bittorrent or rsync or some other protocol which allows correcting a bad block in transit. Users should download a small downloader program which speaks whatever protocol, then the downloader downloads the actual game. (Think about the way Blizzard does WoW updates.)

Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this?

You should also consider using a more modern compression algorithm like bz2 or lzma, if you have control over the application running on both ends of the connection. I'm guessing you'll see these results with Zopfli:

   gzip -c9 > zopfli -i1000 > bzip2 > bzip2 -c9 > xz   # compressed size
   gzip -c9 < bzip2 < bzip2 -c9 < xz, zopfli -i1000    # compression time

Not really sure about xz vs. zopfli for compression speed, but I'm sure that xz's LZMA algorithm gets way better compression than gzip for most data, due to fundamental limitations in the design of gzip's format.

The only real reason to use Zopfli is when the client only knows gzip compression and you can't change it (for example, many, many people have static webpages, CSS, JS, etc., which need to support several different web browsers as clients).

Sounds like we'd be better off using lzma altogether.

> Checking blocks should theoretically possible using HTTP transfers with byte ranges, if you have a separate index file with checksums for each block. That way you get failure recovery but serve your content from Really Efficient Webservers like nginx or Amazon S3. Anyone know of any projects that do this?

Sure, one example is bittorrent (WebSeeding), among others[0]. Turns out bittorrent is a pretty good format for describing chunks of a file (full SHA1 per chunk, variable sized, if you care), and any httpd with byte-range support can fetch arbitrary bittorrent chunks of files.

[0]: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding

I absolutely agree that improvements could be made to our patching system, so here are a few more details as to why we do things how we do them.

1) It is actually working right now. This is the biggest deal at a startup. It's good enough. We have lots of other features to work on that will give us higher bang for our buck.

2) We wanted to be able to use an "off the shelf" CDN for distribution. It's nice to be able to farm out parts of your infrastructure you don't want to deal with.

3) We don't want to have to deal with versions in the patching system. Our patching systems first use is right in our office. The same deployment infrastructure that deploys the game to production runs in our office with every commit from a programmer or artist. When someone runs the game client (or one of the in house tools) the patching system syncs up the files to the latest version.

For real production clients, there can be several versions between the last time someone ran the game client and the next time. It's easier conceptually to deal with.

4) The client is over 100,000 files. Each file is generally quite small which limits the damage of not using an actual diffing solution. The larger files like textures and audio don't diff very well anyway when artists make changes due to the nature of the data formats.

5) It was really fast to develop. It's just libcurl. Once again, we are a small startup so we have lots of things we need to be focusing on that are not our patching system.

All the files are checked against hashes after download so we can correct errors. This is especially important due to the surprisingly large number of buggy caching proxies that some users seem to have in front of them.

One day I would love to make the best possible patching system. Unfortunately it's just not as high on the list as so many of the other things we need to be doing!

From a user perspective I wish you guys would switch to something better like bit torrent (at least offer an alternative torrent download).

I almost gave up before playing the game when an interrupted download meant that I had to start again, and it is pretty frustrating not even maxing out my slow internet connection at peak times for patches.

The fact that you don't have any kind of diffing solution seems like absoulte madness to me.

You don't have to start again. It's just that the progress bar starts from zero again. If you download 2GB of 4GB initially then start again, when you restart it will show 0% of 2GB to download.

It's effectively a diffing system where the block size is on average the size of a file (about 42kb).

Nitpicking my own comment: Actually bzip2 and bzip2 -c9 are the same thing, since -9 is the default for bzip2.

What about the possibility of simply sending the gzip version until the zopfli version is ready, then switching over? Seems like you'd get the best of both worlds.

Have you considered a newer compression program like xz [1]? The backwards compatibility of Zopfli is only necessary for clients that are relatively hard to update, e.g. browsers. If you are already distributing a custom game client, you could bundle an arbitrary compression library that has a better compression ratio, faster compression time, etc. than gzip.

[1] http://en.wikipedia.org/wiki/Xz

I suspect compatibility with deflate is the main factor.

There are a number of LZ related compression algorithms with different properties - speed, compression, suitable to embed, etc.

It wholly depends on what the main goal is, in this case it seems better at rest storage for data that doesn't change regularly is key.

You could do a gzip patch, and then replace it with the Zopfli patch 2 hours later...no?

The last incremental patch we did to the game took 79 seconds in our build system to compress with the usual gzip. This is only compressing the files that were changed since the last patch, not the entire game client, so larger patches would take longer. If we changed to zopfli, it would take 2.2 hours.

Why are you not using bsdiff? This seems like a bug.

Game assets don't change in the ways that compiled binaries do, so bsdiff doesn't sound like it would be appropriate.

Try 7zip's gzip output in "ultra" profile (mx9) - it's within 1% of zopfli and a fraction of the time to compress.

Did you get noticeable compression from zopfli though?

How much time would it save your users, collectively?

Apart from what others said, perhaps faster downloads also would lead to better user retention.

Speaking as a customer of theirs, it's already impressively fast. :)

Why not used BitTorrent?

For example, Facebook uses it to deploy their server code.

http://torrentfreak.com/facebook-uses-bittorrent-and-they-lo...

I'd just like to say that your team's patch delivery mechanism/speed is ridiculously impressive. I've been playing since early closed beta and the way you guys runs things makes me just plain happy. Kudos!

It will be nice for your customers to have a smaller download. To speed this sort of thing up you can probably spin up an elastic cluster to do the compression work, and shut down when it's done.

You should send compressed DIFFs only, and not whole changed files.