Enhancing Photorealism Enhancement

I completely agree with you. Also those people who complain about the "dull european" look, did you scroll down and look at the images altered using the Mapillary Vistas set?

The cool thing is that this gives game-designers an extremely powerful additional tool. You could for example easily go for a more retro-look by training with kodakcolor images ...

And I don't buy the argument that GTA doesn't look more realistic because that's what the designers were going for. If this was a game with a clear "comic appearance" that might be true, however I would argue it looks like it does, because the designers did the maximum that they could achieve with their tools and within a certain budget.

> You don't want your games to look realistic?

The creators aren’t necessarily aiming for photorealism. They want their games to look good enough and be entertaining. Adding more realism could make it look worse. Obviously if AI could solve the hard problems like global illumination that’s great. And I’m sure it’ll be possible to e.g easily add more variation to textures by using larger datasets. But changing the tonemapping to a colder palette, or just replacing the asphalt textures with smoother ones?

The creators of GTA didn’t use the sunset tones in the game by some creative mistake or lack of computing power. They obviously went for non-photorealism deliberately! It’s a stylized view of California. It’s like the burger in the commercial doesn’t look like the real one - it looks a bit unrealistically perfect because that’s actually more attractive than the real thing. Realistic and photorealistic are tangentially related.

The comparison here should perhaps be between frames of the game that already attempted to look like the cityscapes dataset.

GTA obviously doesn’t have a ground texture that isn’t smooth because it wouldn’t be possible, or “too warm” palette because it’s too computationally expensive to have the bleak cityscapes tone mapping.

A GTA frame rendered at its current frame cost, with textures the same size (but different) and a customized tone mapping, would look more photorealistic and make the difference to the cityscapes dataset much much smaller.

The authors could likely have done this comparison since they probably have the ability to modify textures and shaders to set tonemapping. That would have separated out the differences that are only artistic from those that are lacking photorealism because of computational expense.

Actually, I'm pretty amazed by the results. This is indeed a giant leap. Also it's coming from Intel instead of AMD or Nvidia, which makes me hopeful about its wide availability.

The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look.

Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.

> You don't want your games to look realistic? Go play pacman or outrun.

That's a silly dichotomy. We don't even want films to look completely realistic, that's why CGI exists in the first place. Imagine that you ran this over something like an Avengers movie, or even better, Blade Runner 2049. The AI would remove all the stylized/unrealistic details, and someone here would claim that this is clearly an improvement, and any criticism must come from luddites!

This is an enormous leap. Texturing and lighting are two very hard problems in CGI. Realistic lighting using raytracing requires a lot of radiosity passes, which means adjusting lighting of each surface to account for each ray of light bouncing from surface to surface to surface to surface like a 3D billiard ball, more bounces computed yields better results but there’s computational limits to the number of reflections per ray, and many parameters and formulas to choose from each with its own trade offs. Texturing and shader pipelines is another matrix of trade offs to get a good-enough result. Instead of all of that complexity, applying ML this way cleverly hops over the most difficult parts to a really impressive result.

Some don’t want games to look this way. Nonetheless this technique can be applied to give games and movies a different look that would very hard to acheive by standard rendering pipelines, and it doesn’t have to look washed out like this example, that’s all based on the training data set and how much this ML lighting and texturing is blended into the scene, and that’s another creative choice. I can imagine filmmakers wanting to adjust CG lighting and texturing of a scene using a training data set from old TV shows to make a convincing reboot of a classic TV series, or new series done in that style.

Maching learning is often oversold as a shortcut solution for some very hard problems and this is a perfect example.

What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.

>These images are within a hair's breadth of being indistinguishable from reality.

This. I thought those were photos they took somewhere that looks exactly the same as GTA V, but then when I watch the video explaining it I was like holy crap. No Ray Tracing or whatever tech demo was ever this close to photo realistic.

>You have no idea how much cheaper this technique is.

Could someone explain this in greater details? I suppose you will still need to create some decent base image or graphics for the enhancement to work to its maximum?

I think photo-realism in games is not merely a black and white, positive or negative. There are trade-offs, and I think it has to do with perception around aesthetics.

(also, this is simply one guy's opinion, but I am curious if anyone else feels the same)

For instance, in my view, I tend to prefer a stylized less hyper-realism in games. I think this is because the enhanced color, light, and brilliance is more aesthetically pleasant to look at, even though it is certainly more uncommon in the real world.

I don't believe this is because saturated color, and brilliant light doesn't exist in reality, but these occurrences happen more rarely and in specific, often planned, situations and settings. And some people (trained or gifted photographers, for instance), have learned how to capture this look, or to take something rather ordinary, and capture it to appear extraordinary, much like the difference between your average person snapping a photo with a disposable camera, and Ansel Adams. The former will produce a photo that is far more common in appearance, and thus...boring, or uninteresting. So when it comes to games, if you simulate the "average look of reality" the game certainly appears more photo-realistic, but may also be less aesthetically interesting (to me, at least).

This says absolutely nothing about the technology developed here, which is incredibly impressive. Nor do I think there is a right or wrong way create game visuals. I think that misses the point.

However, I feel I'm not the only person who can be blown away with how realistic something can be made to look, yet ...also find it more aesthetically drab and boring to look at. And in the case of games, as I mentioned above, I tend to prefer to see things that are rare, interesting, and extra-ordinary, and thus less "realistic".

Anyone else feel the same?

> These images are within a hair's breadth of being indistinguishable from reality

Without a comparison to the original pictures, they do look very good, the roads in particular look more realistic, as though taken from a dashcam or medium quality phone camera. However the foliage just looks blurred.

A lot of (interactive) CGI already looks more realistic than this. Take for example, this Australian scene made in Unreal Engine 4 - https://www.youtube.com/watch?v=Pg75bfkegtU

And that's not just pre-rendered, here's someone exploring the same scene interactively - https://www.youtube.com/watch?v=y87QL7IPE34

> You don't want your games to look realistic? Go play pacman or outrun.

I want my games to look cool. I dont wanna them to look like streetview or geoguessr.

The result is impressive, but I'm not sure it would be good for video games. Maybe with some added filters it would be a lot better.

While I agree that the quality of this method is impressive, there are two caveats:

1. We do not know the set of pictures that the authors did not show us. So maybe this technique works really well only for a small fraction of cases.

2. Games (especially games that involve moving cars) tend to simplify mechanics so much that the moving end result will never look realistic, regardless how perfect the still picture is.

So yes, I expect great outcomes from such a method, but I don't think its going to be a revolutionary silver bullet.

Not only am I extremely impressed with the current realism they were able to accomplish, I suspect that others will figure out what it is about the final result and that will feed back in to even further realism in games (just like the breakthrough of subsurface scattering in digital skin).

As well, I’m very excited for the scrappy devs who will turn this technique in to something wildly unexpected.

I thought the video was from real life at first until I listened to the explanation. It's amazing that this is possible in real time.

Could they do this process at video game speeds though? (25-60 times a second)

I think this technique has a lot of potential, but will probably need a lot of work before it can be applied to video games. It might first show up as a filter for 'photo mode' in some games though, that would be interesting.

I think you’re absolutely right. The innovations in deep learning make it such an exciting time to be an indie game developer. What used to take armies of content creators can now be a tiny team. Unity should be all over this kind of tech.

Yeah the foliage is remarkably life-like and that is so hard and labor (art) intensive to do well. This is an amazing tool for indies without tons of resources.

> You don't want your games to look realistic? Go play pacman or outrun.

I can't just play GTA 5 as is?

Photorealism shouldn't be the end goal. Watch a B/W television for a week and you won't be missing the colors. Photorealism is much like sugar in that it is addictive, but you won't miss the sugar if you didn't eat it for a while.

Whenever there is a sanctimonious comment like this at the top complaining about all the negative posts, I usually agree with the other posts. As in this case.

I actually don’t think this looks any less “like a video game”, but its color is more washed-out, and darker. It seems like a net loss to me, sorry. Not for fancy theoretical reasons, I just think the GTA looks better in all the examples.

I found their approach to be more like the difference between a digitally filmed movie (that can look too crisp) and one shot on cellulose that has a kind of softness to it.

Where it noticeably fell short was displaying red lights (tail lights, traffic lights, etc) that lost their brightness.

yeah, games visuals are more about art direction than search of realism

this would work great in a flight sim tho, albeit blur and out of focus effects are terrible for visual clarity

however, there's one small bit that has not been given enough attention imho, which is on the latter set, under "removes distant haze": this would be a great technique to complement flat LOD reduction of open world games while keeping up the crispness and contrast of distant items.

That's mostly a problem with the input data. A gamestudio of the size of rockstar could afford to acquire their own custom fit dataset.

Get some good cameras and lenses and capture footage of wherever they want to set their next game. Then they can apply some motion picture color grading to that and use that as the input data for the NN.

I think photorealism has the advantage of maintaining a cinematic feel. You can do a lot of 'realistic' things withs light in a game, that aren't really realistic (you'd never see them in real life), but are the type of thing you'd see in a photo or movie. This allows for a more tightly controlled experience.

Of course, this all depends on what the intention behind the game is. If you do want maximum immmersion, visionrealism would be the way to go.

I don't think you could be any more wrong.

Enhancements like this are going to be the thing that brings VR fully into the mainstream. It will be industry-changing.

This tech is in its infancy. I promise you, the final result is going to be indistinguishable from human vison.

This could extend beyond games easily. I can see adoption from animation studios for eg.

Realistic renders with raytracing assumes accurate models. In the scene, the grass can be a rectangular block with a grass texture applied to it for instance, NN will transform it into an actual grass patch (see example images) similar to how it would look like in real life (not constrained by your 3D geometry / scene setup). So realistic lighting is just one part of the problem space here. A road with an asphalt texture applied on it will need a lot of manual tweaks / material design to look realistic under raytracing regime, NN can look at it and implicitly say "hey, this is a road, and I know what a road looks like under this lighting, so I'm going to render it like this" and it automatically becomes realistic.

> [...] Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed.

Define 'photorealism'. You limit yourself to the rendering aspect and ignore the content/asset problem and thereby economic factors.

This technique is probably one or two orders of magnitude cheaper than generating geometry and shaders that have the required level of detail in any traditional way.

Either by an artist or programmatic/procedural (Someone has to write that code too/set up that node graph in Houdini or the like). Yes, you can also just 3D scan stuff (see Quixel, etc.) but that has limits too.

More specifically for achieving photorealism by 'traditional means': consider scales.

An asset, however produced in finite time, will only hold up to some range of scales. The technique in the paper allows to get as close or far away as you want and everything will still look real. No level of detail work needed, just a few more training data.

Traditionally, if you use textures, you will reach the limit of the texture resolution. Even if you use procedural techniques – not everything is a fractal. A close up of a brick looks very different from a wall of bricks etc. Again, who will write that code?

My guess is rather that we will see a hybrid of more photorealism by traditional means and more 'icing on the cake' by methods like this.

I think it's worth exploring multiple approaches to a problem. Maybe this neural network approach can be improved upon and it will also generate photorealistic looking images but have other advantages such as less effort on behalf of the game developer or less compute.

The Cityscapes dataset is a bit murky looking; try scrolling down to the Mapillary Vista dataset instead. Much better colors. They should have led with that one instead.

Good ray tracing is expensive and still requires a lot of asset preparation. Difference between what RTX can do realtime right now and production quality final renders is massive. I think there will ultimately be quality wall for realtime tracing that target ms/frame versus mins/frame for HQ renders. After which hacks like this must be used to further close distance by adding layers of additional details that really sell photorealism. TBH I doubt RTX could reproduce even this quality for many years, if ever.

It's not just about murkiness and lighting quality, this is akin to adding the last 10% of polish that usually takes 90% of the time. It's also the difference between amateur vs much greater proficiency level. Having access to the assets of scene in 3D max, someone with a basic training can replicate base GTA scene, the enhancements shown here takes years more experience. The amount accessibility and saved labour is tremendous. It's why even VHS video is still more photo realistic than 4K RTX. Real life has a lot of minutiae and interacting details beyond resolution and good lighting models.

I watched their video, and I tend to agree. With the amount of computing they use, ray-tracing is probably going to be just as good and probably higher frame-rate. But maybe this technique, in due time, will result in an alternative class of rendering pipelines with less focus in shading and materials.

You no longer need 3dsmax for raytracing. But what could be really interesting is to train your ml model against path traced images and apply the outputs to selective areas of your screem where the g buffer provives insufficient information density (i. i.e.foliage)

I don't think the goal of this is to make games look better.

Especially the GTAV dataset is often used as a synthetic dataset for research. The appeal is that you can extract normal/material and other semantic information from the game engine for a driving scene.

Basically avoid expensive segmentation labels on real data by working the other way around. Things like this synthetic to real translation can then be used for other downstream tasks (semantic segmentation, distance/normal prediction).

I share your disappointment. There was recently a thread on HN on how good metadata always beats machine learning.

I wonder how much complexity could be saved by letting the application properly annotate objects: "This is a tree located at coordinates X,Y,Z in the scene. Please NN take over and project a tree appropriate for the light conditions and geographical location."

Good point. But I wouldn't rule out neural networks. They could still be used to derive efficient shader models that can be run in real time. https://arxiv.org/abs/1603.06078

For an example of photorealistic rendering approach coupled to a scenario generator, see https://synscapes.on.liu.se/index.html

This is what has been called Deep Shading: https://arxiv.org/abs/1603.06078

There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( https://youtu.be/P1IcaBn3ej0?t=462 )

Is it real-time rather than post processed?

Part of the contribution of this paper is about preventing that specific problem.

“We also seek to eliminate artifacts that can be seen in the results of prior deep-learning approaches, which often hallucinate objects. To this end, we analyze the datasets that are commonly used for photorealism enhancement. Our analysis reveals that their scene layouts differ in ways that can explain artifacts commonly seen in prior work. To better align the datasets and alleviate the artifacts, we propose a new strategy for sampling image patches during training. We further design a new adversarial training objective that facilitates enhancements that are geometrically and semantically consistent with the content of the input image.”

From what I can tell, the whole reason the road "looks like a video game" is that it's using a source texture and then stretching it along the road's geometry, "diluting" the level of detail as seen from the camera viewpoint.

The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are.

Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road.

If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.

This is a good question - it certainly is possible to run a NN on the artwork or pre-g-buffer, and you could imagine having something bake the changes.

To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild).

So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer.

Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why.

That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.

That's because of the dataset. Later in the video they show results using another dataset with more vibrant colour, that looks a lot better IMO.

The model trained on TSIT gives nicer results.

Probably yes. Alternatively, it shows that the road textures used in GTA V were not very apt for their job. I do think the smoother ones look considerably more realistic.

For me the most unrealistic part of the GTA renders was the trees, and the post processing significantly improved that. Blur is also an important part of real photos, it's unusual for everything to be in perfect focus.

I would argue that these kinds of renders are getting good enough that we should be more precise about what is meant by photorealism. Most pictures i take with my phone have alot of blur, don't capture colors well, and are frankly quite crappy. I feel like people expect photorealism to be equivalent to a really well-planned photo by a professional.

There is a long an rich tradition to release your DNN/FX work in a post stamp sized presentation.

Style transfer commonly hallucinates objects or adds artifacts. Making something look like a painting is actually much easier than making it look realistic. In paintings, artifacts are often tolerated as 'artistic'. In a photo, it just pops out. Also, style transfer approaches are not temporally stable most of the time. Have a look at the comparisons with state-of-the-art image-to-image translation (CUT, TSIT) and photo style transfer (WCT2). In the case of GTA/Cityscapes shown here, most of these methods put trees in the sky and/or flicker. Also the use of G-buffers allows a much deeper and more robust translation than just using images as style-transfer approaches.

It's also giving the license plates the blue stripe.

FWIW, I think it doesn’t compare to PBR - these are two completely different things, and there’s no reason you can’t do both. Maybe as a simple example of that, it’s common today to run a NN based denoiser on top of a physically based path traced rendering.

This neural network operates on images - input is image and output is image. Note that it cannot synthesize an image from scratch, it can only take an image and make it look more like the lighting in it’s training set.

PBR is a technique for taking a 3d scene description and synthesizing an image, so for PBR you need models and lighting and materials, and even with physically based techniques it might still come out not completely realistic looking. There’s always room to emulate realistic subtleties of texture, lighting, camera lenses, video/film response, outdoor colors, atmosphere, etc.

For one, it will be a lot faster to compute. For two, it is a lot less laborious work to set up, which does not scale with the size of the world like PBR, but only with performance.

So where previously, you needed a lot of designers to make a big world look realistic with PBR, now you need to collect a big dataset with the style you want to have, and apply it to your game. I suspect that will scale differently.

You mean the shadow of the tree?

Is the windshield glass really the only change you see, when grass becomes green, reflections are added, the entire photo has a higher color temperature, etc?

I think the Cityscapes dataset pictures were taken from the car's hood; the Mercedes emblem is up close. The camera position in game also seems to match this.

> GTA feels like intern 3D rendering work

Lol, I guess you’re new? Or maybe you don’t know the difference between 3D rendering software and game engines?

GTA V came out in 2013 and the fact that it is still used as a reference for graphics quality tells you everything you need to know.