Screen and asset image scaling for games | Hacker News Reader

Screen and asset image scaling for games

coderofworlds.com

65 pointsspeeder26 comments

Some time ago, I wrote some articles on image scaling that go into a bit more technical depth (with a specific focus on mipmapping):

http://number-none.com/product/Mipmapping,%20Part%201/index....

http://number-none.com/product/Mipmapping,%20Part%202/index....

A problem I get into at the end of the second article is that gamma-correction is very important for good image scaling results. However, almost nobody gamma corrects during scaling, even today.

I might be misunderstanding, but shouldn't gamma correction happen after drawing your scene?

You are thinking of a different gamma-correction step. The reason you gamma-correct during rendering is because monitors expect data to already be gamma-corrected, and it is this expectation that causes the problem I am talking about. Data in a bitmap is not linear with the light-intensity of the colors; it is gamma'd!

The simplest kind of mipmapping is a box filter, where you are just averaging 4 pixel values at once into a new pixel value. Thinking just of grayscale pixels, if you add 4 pixels that are each 1.0 (if you are thinking in 8 bits, 1.0 == 255), and divide by 4, you get 1.0 again. If you add two pixels that are 1.0, and two that are 0, you get a value of 0.5. Which would be fine if your bitmap were stored in units that are linear with the actual intensity of light; but they are not, because they are in a weird gamma! What you are going to get is something like pow(0.5, 2.2) which is way too dark.

Thus when you don't gamma-correct during mipmapping, bright things lose definition and smudge into dim things way more than actually happens in real-life when you see objects from far away.

> Data in a bitmap is not linear with the light-intensity of the colors; it is gamma'd!

That is true for common sRGB images, such as what you would see on the web. But for games which usually have tightly controlled art pipeline it would not be unfeasible to use linear colorspace bitmaps as textures, and in such case you would not need to have gamma-aware scaling.

It is somewhat infeasible to use linear colorspace, because you need a lot more precision in order to do this without banding. You end up with substantially bigger texture maps, possibly twice as big; but actually it ends up being a lot more than twice as big, more like 8x or 12x, because the compressed-texture formats that graphics chips are able to decompress onboard do not support this extra precision. So if you were to try using something like S3TC compression to reduce this bloat, the result would be really ugly.

In general, games only use light-linear texture maps when they also need HDR in the source texture, which is not that often. Ideally it is "purest" to use HDR for all source textures, but nobody does this because of the big impact on data size. (And even for the few textures that are stored as HDR source, often the data will not be integer, but some other slightly-more-complex encoding.)

[Claimer: I have been making 3D games professionally for 17 years, so this information is accurate.]

> gamma-correction is very important for good image scaling results

This is a great observation.

When offering upscaling options, it's great to offer all the things mentioned in the article. But please always offer nearest neighbor as an option too. I want to see the original pixels as close to their original form as possible, and so do many gamers. Pixel art is great, the "blockiness" is part of its charm.

That's only true for PC games, where monitors did indeed produce blocky pixels. Arcade, console and early computer games that connected to TVs were designed to take advantage of the "deficiencies" of those screens to produce smoothed images, or the illusion of more colours than the systems were capable of, among other effects. So if you want to play those games as they were meant to be, you either output directly (without any upscaling or filters) to an old CRT TV, or you use those NTSC filters or shaders that most emulators have that try to mimic (to various degrees of success) the way an old TV displays images.

I disagree, and so do most/all retro gamers nowadays. The golden standard back then were the standard res RGB monitors that arcade games used. Consumer TVs in America at best had composite input and paled terribly in comparison. But we all dreamed of our games looking as crisp as they do in the arcade. Even the NES had an arcade counterpart where you could experience NES games in crisp, beautiful RGB.

Standard res RGB monitors are not as crisp as PC monitors. But they are still quite nice and really do let the individual pixels shine through.

Most retro gamers now typically hook up their consoles via SCART to a CRT that accepts pure RGB, typically a Sony PVM. I've got all my consoles and my arcade boards running on a PVM and can really easily see the pixels making up the art.

here's a few examples I grabbed off the web: http://imgur.com/a/GXHyf They look even better in person.

Yeah most emulators provide filters that give you scan lines and all that jazz (none of them look quite right). And of course an RGB monitor does not look at all like an old game running in a modern emulator with no filters. But I would still definitely say you can see and appreciate each individual pixel.

You remembered me of some stupid browsers that insist in filtering the image, making some specialized pixel art sites look absolutely horrible.

There's an upscaling algorithm he missed: http://research.microsoft.com/en-us/um/people/kopf/pixelart/...

I don't missed it, I ignored it on purpose, for two reasons:

One, I did not understood it well enough to try to explain even the basics.

Two, I could not find a binary or source I could compile easily to make a example image.

Also you can put a "three" there: it is not widely used, while even the recent xBR has a couple of plug-in implementations on emulators for example (and has binaries)

EDIT: Lack of binary is also why I did not made sections for 2xSal, SuperEagle and so on... If people help me finding that kind of stuff or creating the needed images, I will gladly update the article.

This looks like a FOSS implementation of Depixelizing Pixel Art - https://github.com/jerith/depixel

That's probably my favorite research paper ever. The web link in the PDF is offline, but it looks like they moved everything to the Microsoft site -- http://research.microsoft.com/en-us/um/people/kopf/pixelart/. I just spent a good 15 minutes mousing over the image comparisons again

One alternative upscaling method (for non-integer ratios) is to first upscale with no interpolation a integer ratio to make a bigger image than the target size. Then the large image is down-scaled (with some suitable algorithm/filtering/interpolation) to final size. Afaik it is what Apple does with it's "Retina" MacBooks and thus I have dubbed it "retina scaling", but there is probably a proper name for the method too.

edit: comparison http://imgur.com/a/fC8iQ#1

In a class I took on image processing, this was offered as the definitive way to do non-integer scaling. Apple certainly didn't invent it.

While not really general image scaling, there's a lot of work being done on post-processing in emulators to try to emulate CRT displays (like the aperture grille) or even the quirks of NTSC.

http://emulation-general.wikia.com/wiki/Shaders_and_Filters

Heh, it mentions that noone can emulate yet shadow mask CRT.

Those are what I mention in my article (TVs that have triangle pixels composed of 3 circles)

I never use current CRTs emulation because I guess I never saw TVs that could be emulated, I think all CRT TVs I saw consoles running were of shadow mask type, having almost no scanline (and I was a curious child that used to put amplifying lenses in front of TVs to see their pixels...)

A great way to deal with scaling assets for mobile is to create your art in a vector format to begin with. For Uncle Slam on iOS we did all the artwork in Illustrator and exported to PNG at three sizes: iPhone 1x, iPhone 2x (and iPad), and iPad 2x. The results are sharp images with the right level of detail and no scaling artifacts.

We even took the exact same artwork and made an 8 foot tall banner with it, and it looks great!

You assume here that size doesn't matter, when it might. If the difference is only initial download size in your game, and is a few dozen MBs, then yes I agree. but for some applications, this is content that is loaded in runtime, and since size DOES matter in this regard, some applications will need to upscale images.

On Kidoteca some of our art is made that way, but not all of it.

Our backgrounds for example frequently feature "digital painting" style, with stuff that is outright impossible to do in vector.

What we did for Return to Dark Castle (which ran at a fixed 640x480, 16 bit color) is give the user the option to scale by nearest neighbor, interpolation, or both (mixed 50% with each other). The "both" option worked surprisingly well, especially since it was a "quick fix"!

The reason linear filters don't work for upscaling, say a Lanczos or Sinc, is that the base supposition made for generating the filters is that the source is "bandlimited" -- that is, all images are samples of some source which can at most vary once every pixel. Problem is, while that works very well for regions that are supposed to be smooth -- textures and stuff -- it fails at edges, since edges have infinite bandwidth. The result is a wash out as you can see for linear/bicubic interpolation.

Of course, it does work almost perfectly for downscaling, because in this case we're not creating any edges, just reconstructing the original edges -- an those vary at most once every pixel.

Actually, the effect happens just as much during downscaling. It is just not as noticeable, because your artifacts happen at subpixel scale. But they do affect the output colors.

The math does not really know the difference between upscaling and downscaling.

You run into pretty much exactly the same set of problems in the audio DSP realm whenever you have to deal with audio of different sample rates or with variable sample rate algorithms like tape delay emulations.

Most typically in that domain you also use windowed sinc filters and there's a ton of literature on the tradeoffs of specific window designs, as well as very fast fixed point implementations etc.

It's all pretty interesting stuff and trying to make it run efficiently on modern mobile hardware is a fun challenge.

I'll add my 2c with an article that I wrote some years ago. http://www.screamingduck.com/Article.php?ArticleID=35&Show=A...

It's a look at kind of a interpolation weighted by scale2x.

Reply on news.ycombinator.com