Google Stadia uses Style Transfer ML to change video game art in real time | Hacker News Reader

Google Stadia uses Style Transfer ML to change video game art in real time

turingtribe.com

143 pointspavel_tiunov91 comments

To me this was the least interesting part of the Stadia reveal. State share was the real innovation imo. Would any serious game studio use style transfer ml to actually replace their visual artists? Didn't seem like the quality was there.

That said, it is an awesome PoC. Just not something I see being practically applicable.

> Would any serious game studio use style transfer ml to actually replace their visual artists?

No, a serious (AAA) studio would not. OTOH, the platform might intend to support producers that aren't AAA studios, and whose limited pre-release resources are focussed on gameplay more than art assets, and those producers might view it very differently than AAA studios.

You still have to do all the work of building the geometries for the art assets, are you really saving that much on not having to color/texture those assets?

Maybe? One of the hacks of adding more assets without having to do much work is just recycling existing assets by assigning different color schemes/textures.

That's one thing, plus look at the result. This makes a very lowpoly game quite impressive visually.

And this is just a trivial graphic filter through a neural network. A lot more can be done with AI.

Come on, think bigger. If they're doing this for art styles you can bet your bottom dollar they'll be doing something using Google images. No longer having to actually create 3D meshes - just a Google image search, click a button and have the geometries automatically created for you. Then just add an automatic style.

>> Come on, think bigger.

True.

Actually Google is just seeing the wealth of innovation in procedural generation and in computer creativity, knowing where this is going, and just want in on the action.

And since as a cloud gaming provider, they stand to make much more money than tool vendors, their incentive is much bigger.

Indeed, I'd be curious to see what some indie devs do with it.

Imagine a game where you step into a portrait and are suddenly in the art style of the image. You suddenly walk into a countryside that feels american-gothic style, and then time-jump to 1940s New York in the style of Nighthawks.

(different) Stylization as a medium in games isn't that explored, and the opportunity for indie devs to play with that is, IMO, really cool.

The Witcher Blood and Wine expansion actually did a great job with this. There is a main storyline quest where you enter a haunted painting

Would it help in game development? For example giving developers better looking playable prototypes while the artists are still working out their designs?

Any technology that can automatically perform the tedious task of coloring 2D art would be a godsend to indie game devs as well as amateur artists.

Hypothesis: the rough and relatively unpredictable results are not so much a good fit for finished products as it is for _rapid visual prototyping_. A good art director could use this to quickly settle on a rough goal around which the rest of the game would be targeted. I'm aware the pipeline usually goes from art director -> development, but this could help, say, a competent dev team with a gameplay direction but no artist yet, or a long-standing developer that wants to use its existing engines and mechanics to draft new, distinctive franchises.

This is the use case they highlighted in the keynote. It doesn't appear to be a system designed for shipped games.

I think the more interesting applications will come when you get tools like this used by visual artists, complementing handmade design. The demo is cool but uses style transfer very bluntly, applying one style to the whole image with high intensity. I imagine the future of this is something more like neural "shaders", able to apply real-time pixel alterations to scenes or objects within scenes. Used carefully, they could increase the range of creative choices a visual artist can make.

This is where I think the value lies.

I have been thinking about a process where a procedurally generated template is completed by artists and subsequently used as a base for training an AI to apply to further procedural generations.

I also wonder if you can do micro style adjustments to unify artistic style, train a net on one particular artists work, the rest of the team copy the general style but use a final pass of a style transfer to make it a closer match.

Honestly seems more like a user-end personalization than something a developer would use. In a game settings menu, you choose an image that maybe you uploaded to use as the game style? Definitely something people would use. Especially if they figure out how to make it selectively apply to only certain textures/objects or parts of the screen, which seems likely.

Street Fighter 4 had a feature kinda like that (with predefined selectable styles). AFAIK it was not popular. Sometimes my friends would turn it on as a joke.

Exactly...This is a cool gimmick to try once, but it will seriously distort the intended art style.

It is another example of forced intrusion of DL techniques creeping into places where it is not needed in the first place.

Gimmick is the perfect way to describe it. I really don't understand why they even included it.

Actually, I wonder if there's potential in something like this for accessibility purposes. If you have one of the many variations of color-blindness, for instance, could such a restyling be helpful? Could other neurological disabilities be similarly aided by dynamically restyling visuals?

There are many low-tech solutions that can be applied instead of shoehorning in neural networks for this purpose.

Yes, but they also require active participation from the developer and many developers are unwilling or unable to spend the time to implement them, especially when there's a rather large range of disabilities to account for.

It's possible that tech like this could offer a user-definable solution to these issues.

One incredible use is actually in network training for things like AR and autonomous driving. You want a slightly fake world and you could easily try out various network approaches using a free fake world found in game engines. Microsoft has something for this but cannot find a link or the name right now.

I would have been much more interested in using deep learning to make human faces more realistic. Somebody was experimenting with it in the past quite successfully IIRC, but I'd love to try it on a real game.

Maybe used as a player entering dream like equivalent after eating magic mushrooms. Other than that you’d be hard pressed to see this in any game in memory

> Would any serious game studio use style transfer ml to actually replace their visual artists?

I doubt that was the point. IMO it is more of a palette testing tool.

I was baffled with the style transfer announcement. It seems arbitrary, random, and out-of-place. Style transfer (applied to art) is really neat, but I can't see it being used to re-clothe games for production.

Before I get downvoted, yes I do think it could be useful for prototyping and conceptualization during development, but I doubt anyone would actually ship a complete game with the entire camera view transformed by style transfer.

A neat tech demo, but as with so many things coming out of The Valley these days, I don't think anybody actually wants this. No actual game is so bland that you'd want the ability to overlay your own "themes" on it after the fact. AI has lots of potential for assisting artists, and maybe part of that will include post-processing effects, but putting such things in the player's hands is pointless.

this is certainly just a neat gimmick, but it's interesting to me for a particular reason: the Stadia box uses an AMD GPU, and per the Google announcement, all the style transfer is done in real time.

this suggests that we may soon have better AMD support in TensorFlow.

Style transfer is most likely not running on the GPU in this case. It's probably running on Google TPUs, given that the GPUs are likely already under high load in order to play AAA games at 4K HDR 60fps.

where can I find out more about this? all I'm going by is them saying in their keynote that its running "on the Stadia instance".

Tensorflow support on AMD is supposedly not awful anymore. https://medium.com/tensorflow/amd-rocm-gpu-support-for-tenso...

I wish someone who had >actual< practical experience in the domain would comment.

It's an important question.

I haven't personally tried to do anything on an AMD GPU, but AMD actually has a very active GitHub repository https://github.com/ROCmSoftwarePlatform/tensorflow-upstream .

If you go through the issues you can get a sense of what it's like from people who do have practical experience.

this is like bad selection bias though. people never write issues to say things are going well. the general Tensorflow issues are also about only bad things, by definition

I've been following ROCm development closely, and everything I read is pointing towards it being a first class competitor to CUDA.

That being said, I don't have any personal experience because, well, I don't have AMD GPUs to play around with.

Fun, but nobody will ever do it for more than 5 seconds before getting a headache and turning it off again.

Consensus in this thread seems to be that this is a neat gimmick but nothing else.

Maybe because I have a friend working on a video game as a solo developer, and putting a /lot/ of energy/money/time into art--I really see the potential in this (type of thing).

This is not really for the player's benefit to put on whatever mods they want.

This is for fast iteration for artists and creators to automate a huge (sometimes necessary evil) burden.

> Maybe because I have a friend working on a video game as a solo developer, and putting a /lot/ of energy/money/time into art--I really see the potential in this (type of thing).

For that usage Nvidia's thing was way more interesting: https://blogs.nvidia.com/blog/2019/03/18/gaugan-photorealist...

That gets you usable texture assets super quickly, which is probably what your solo developer friend spends the most time on. That or models. Neither of which Stadia's style transfer helps with. They probably aren't spending much time tweaking a shader-applied post-process effect, but hey, maybe they are, and this style transfer ML thing is useful. Very, very unlikely, but maybe.

> For that usage Nvidia's thing was way more interesting

If you're building a photorealistic game. And there's already lots of tools on the market to generate these sorts of textures.

I could see this Nvidia thing used for much better looking mockups / concept art. But it doesn't seem suited for final asset generation.

> But it doesn't seem suited for final asset generation.

Well yes, of course, and neither is Google's Style Transfer ML. These are both just prototyping tools, not a magic wand to make production-quality assets.

It could be that they are throwing it out so that the community try to find a practical application of it. It is obvious that they understand that what they are demoing is not the end product so to speak, but just the best way to represent general idea.

This seems like a solution looking for a problem.

I'm already excited to see what Microsoft will show at E3 with Xcloud streaming service that they have been working on for a couple of years- lots of great progress in that area right now.

I've seen a number of these impressive looking demos..for example: https://www.theverge.com/2018/12/3/18121198/ai-generated-vid...

Impressive sense of realism in level of detail variety..show the potential...BUT consistency, sharpness and realism of object boundaries would not survive close inspection.

This is not a consumer product at all. I don't know how useful it will be but it's designed to be development tool for early art direction visualization.

Even for 2d visualization it could be very useful if you could drop images into your website mockup files and have it intelligently apply color palette and texture information to your existing content.

It won't look good, but it might look good enough to save you from having to manually tweak the design 20-30 times to explore a range of styles.

this can potentially be a "mod" play... where individual gamers can change game "skin" and stream videos on youtube...

I don't think there will be heavy users of style transfer among high budget games anytime soon, but the important part is that it's done in real time. With an assumption that it has a reasonable level of latency, probably AA, super resolution or any other neural net based quality improvement can be a real thing.

The useful application for AI in streamed gaming seems like it would be for predicting the next 30ms to avoid input lag.

This seems gimmicky at best. Prediction has to take place on the client though - so the challenge is making it cheap.

This is really cool to think about. In practice though I would imagine every incorrect prediction to be incredibly jarring. Additionally my intuition is that latency is easier to decrease than predicting the next 30ms. If there is some level of latency that is very difficult to reduce (more so than training an AI to predict the next 30ms with very high accuracy), I'd be curious to know why.

Between e.g. central and west US you'd have a physical ping of ~10ms so 20 round trip just based on the speed of light (i.e. very hard to get rid of). Meanwhile the input lag you want is one or two frames so on the same order. But this lag is additive, when the game receives the new state it has to compute and render the world, the display has its lag before it shows up etc. So the new frame has to start computing as soon as the input is registered in order to hit reasonable input lag.

Predicting 2-3 frames should't beyond what we can do at least roughly. Especially for some important inputs such as turning in first person view it should basically just start moving the screen. I belive John Carmack tried some kind of rotational prediction (Not based on AI) for Oculus.

I am not a game developer, but couldn't you make a multiplayer game more fair in this situation by adding artificial lag into everyone's controls to match the highest lag person? Just limit it to a certain amount, like no more than 200-300ms.

I wasn’t considering fairness here (that’s a whole different can of worms). Even for a game where everyone’s lag is the same, X milliseconds of input lag and Y milliseconds of network latency is painful for some values of X and Y. For fast paced games I’d say X is probably just a couple of frames (15-30ms) and Y is about 100ms.

Silent hill concepts could be fun with this. Otherwise eh. I would like to see Jojo palette swaps occur to highlight tension but I don’t see that being implemented with something like this.

Did anyone else see this headline and think: Why would they use a customized XML for this in 2019?

There is nothing more Googley than building something that's technically impressive with absolutely no - or even antithetical - market fit

Onlive - required other people's infrastructure and a hardware play, and a software client.

Playstation Now - required Sony to buy/use a cloud service, maintain it, and expensive Sony hardware just as a force of habit.

Google - negligible instance running on their existing infrastructure, maintained under existing processes, playable on chromecasts already in hotels and homes everywhere, alongside browsers and mobile devices.

If Google doesn't shut it down within the 5 years for people to notice this is even an option, they could focus more on titles and developers more than any other entrant to this market.

And perhaps the gaming is just the application to get people's attention.

Are you implying Google doesn’t need to maintain an expensive cloud infrastructure for it as opposed to Playstation?

The only reason they can do that today without specialised hardware a-la OnLive is because Nvidia and co. now have specialised GPU circuits allowing one to stream the encoded video output to memory.

When OnLive started they had to do it themselves.

I don’t believe you have any idea what you are speaking about.

> Nvidia and co. now have specialised GPU circuits allowing one to stream the encoded video output to memory.

Could you clarify what you (and the comments mentioning NVENC below) are talking about ?

As far as I understand it, NVENC and the ilk are solutions that capture video output and encode it to H264. So they are x264 encoding accelerators, nothing more.

If you were running a datacenter this way, you'd be much better of having a bank of Matrox capture cards (which support multiple simultaneous inputs) in dedicated hardware converting your video game output to x264 streams for broadcast. They even support capturing, x264 encoding and streaming to IP addresses on your network for further distribution.

What would be interesting is if they bypass the video display output phase completely and render the finished framebuffers to memory (or an encoder chip via DMA over PCI) instead. I assume this would cut down manufacturing (+licensing for HDMI ?) costs a bit.

I know AMD has DirectGMA that allows other devices on the PCI bus access to limited chunks of GPU memory. There are signalling mechanisms in DirectGMA so that devices can basically implement producer-consumer pairs.

As far as I know this doesn't exist in consumer chips though. You need "workstation" GPUs. Which might explain Google's particular GPU choice, now that I think about it.

You:

>> As far as I understand it, NVENC and the ilk are solutions that capture video output and encode it to H264. So they are x264 encoding accelerators, nothing more.

Me:

>> allowing one to stream the encoded video output to memory.

You're being pedantic. What I meant is that OnLive actually needed specialised hardware because they could not provide their service without it (as in: there was no capability in hardware, without changes to the codebase - to render not to an external display). You, on the other hand - are speaking about the conceptualised "perfect" implementation.

Google doesn't need that hardware in order to provide that service because by now there's a hardware capability that allows you to capture the screen of the device you are rendering to. They could try to be more efficient by using other hardware, but that's not a prerequisite for their Gaming Service.

>What would be interesting is if they bypass the video display output phase completely and render the finished framebuffers to memory...

That's exactly what NVENC, Capture SDK, AMD's AMF, and Intel's QuickSync do. Just encode the framebuffer output of whatever game engine you're using with one of those API's. (And remember to make sure your game engine license allows you to use it that way. As always, if you use Unity, you're out of luck, they specifically forbid you from doing this.)

Thank you for this. I was thinking that my understanding of NVENC was completely wrong based on what the sibling posts are positing.

> Nvidia and co. now have specialised GPU circuits allowing one to stream the encoded video output to memory

Can you give a little more info on this? Is this standard to consumer level GPU's? What is the tech called?

I'm assuming you mean how can you encode the output of the nvidia card to a memory buffer using an API?

The NVENC API allows you to do this on your commodity NVidia card. If you have the good stuff from NVidia, you can likely use the Capture SDK, which is a bit more scalable. If you use AMD there is AMF. And if you're cheap, you can just use QuickSync from Intel. All of the encoding API's will likely be unified with an industry wide API a la opengl at some point in the future, but it won't be implemented by everyone in a timely fashion, so you have to pick your poison right now.

One gotcha though, you can only do this if your game engine's license allows. Which means anyone who uses Unity is out of luck. Unreal and other engines? I don't know, I think you can talk to the companies and they'll probably? give you permission? (At least I'd hope they would give you permission until they go into the cloud gaming business.)

Your best shot though is just to take an open source engine, like Godot, and "NVENC it up" so to speak. It'll take maybe a couple of hours and it'll save you a lot of headache.

Nvidia's is called GameStream, and has been in consumer cards for about 4-5 years now. Moonlight[1] is an open-source client that lets you stream your PC games to most any device.

They may be referring to ShadowPlay, a feature that keeps a buffer of the output so you can save something that just happened. Again, available for 4-5 years.

[1]: https://moonlight-stream.org/

> I don’t believe you have any idea what you are speaking about.

You are gatekeeping based on irrelevant details when my point is that prior entrants had higher or untenable costs and Google might not, which I think you agreed on.

Google's advantage is that it doesn't need a dedicated hardware play, has saturation in relevant markets in their existing hardware plays, and has existing capacity in their large compute infrastructure. Whether consumers care is another story, but they can throw more money at things to make them care without having other untenable overhead costs as a distraction.

Google's advantage is that they've got loads of cash. Speaking of that advantage is adding nothing to a conversation.

Where are you getting that it’ll only require a negligible instance on existing infrastructure? Running a AAA game on 4K takes a pretty beefy machine. I doubt it’s a workload that you can throw on some small VM without making any changes to your hardware infrastructure.

> playable on chromecasts already in hotels

Are there actually Chromecasts in hotels? I have never seen one and curious which chains have them.

The hotels I've seen them in (in the US) have been Hampton, Hyatt, and Mariott. I might've seen one in a Sheraton, but don't quote me on that (and if so, only once). Definitely seems to be per-hotel rather than brand-wide, though.

Yeah its a game changer. Marriott brand is rolling them out, and Hyatt has them too. These are massive portfolios now, and will save them from getting screwed over continually by Apple changing the docking station ports.

For now Chromecast just being there and part of the TV will let users do what they want with music and now this.

Throw in a VR headset and you got Ready Player One.

The market is dead because Nvidia only want their high tier graphics cards to be deployed... can you imagine if instead of deploying 30,000 dollar GPUs, you could drop quad Titan X's or Titan V's. The datacenter license is cancer!

Can't wait for ads to be inserted into games with this...

A few days ago I commented on the Stadia on HN and basically predicted the same thing. Does not seem to be a popular opinion however..

I do find HN to be against sarcasm of any kind, which in it's own way is slightly ironic. I enjoyed your comment [1] and consequently upvoted it!

[1] https://news.ycombinator.com/item?id=19433773

That was sarcasm? I though he was serious and upvoted.

I can imagine walking down the street in Call of Duty game or something like Cyberpunk 2077 and seeing ads overlaid on wall textures of buildings.

A while ago PUBG plastered ads for their Berlin Invitational. While they were ads related to the game, the incrementally added them EVERYWHERE!

First they're on a bus-stop wall, that's thematic okay. Then they're on random walls - getting annoying. Then the loading screen and the parachutes, nows it's annoying. Like, guys I'm from the US and I don't want to go to your stupid competition!

Is anyone else concerned this will be yet another Google product that goes by the wayside in a few years? This has essentially become my biggest fear now with any new product they announce/release.

I understand that it's inconvenient when services shut down but I see 3 options.

1) Wait until it seems there's enough comittment from Google before using it. 2) Start using it right away knowing that it might not be around forever. 3) Don't use it because you don't trust Google to maintain it.

Given these 3 options, does Google launching a service really have that much impact on consumer's lives? I'm a huge Gmail user and have been for over a decade. If they shut it down it will suck but I'll find another way to email people.

I get the philosophical arguments but curious why some people have the level of concern or angst when Google announces a product.

(for the sake of this comment let's forget about competition and monopolies)

It's pretty much a certainty with this project, in my opinion. Until everybody somehow has gigabit lines to their home and to their phones, I don't see this becoming popular. Like most Google projects, their actual success period is most likely 10+ years away, and the project will be axed long before then.

5G deployments have already started rolling out. 5G has average latency in the range of 1-2ms. Google controls a huge swath of the backbone after that. They can realistically achieve < 5ms latency on 5G devices in the next 1-2 years.

IMO, a more valid argument against its success would be that any consumer who is in a position to ensure they have fast enough internet for this service, is also in a position to buy a gaming console. But the counterargument to that is that there is no reason someone cannot have a gaming console and also use Stadia.

I'm normally very skeptical of Google's motivations, respectful of their cloud technology, and doubtful about many of their products. But I think this one will be a success, and shortly followed by competing products from Amazon and Microsoft.

Seeing as I live just on the outskirts of a capital city (Cardiff), and I can't even get 3G signal inside my house, and no more than 80/20 broadband; I'll remain skeptical about the possibility of success for this outside of very specific cities. There's also a lot to be said for reliability and the requirement to always have a fast connection: one of the benefits to many videogames is the fact that they are not online.

Isn't 5G on a wavelength that has poorer penetration than 4G? It might work in the USA but signal is a constant issue in stone housing here in the UK (though it may be better where you are based). I had to get a femtocell from EE in order to get signal in my house :\

I doubt this is going anywhere. As a PC gamer I have no need for this, don't want to deal with lag, and don't trust Google to operate the platform in a consumer friendly way.

Look at how angry people are at epic right now. Taking a brand that already has a bad rap and trying to start a new market (that has already failed several times over) is not a great angle, especially with an audience as skeptical and anal retentive as gamers.

Additionally, who is the audience for this? Casual gamers use consoles, and those more invested use PCs. This is too out of the box for the former and completely useless to the latter. I could also see Google trying to get exclusives which would be a PR disaster.

Sure, it's a novel idea and looks like a decent execution, but its by no means a new one. Onlive did this 10 years ago and couldn't get market share. PSnow is available right now and its completely unusable even with a great connection because they dont have enough servers. The game selection is also terrible considering they should be able to load up any game in their ecosystem. I cant play MGS4, the PS4 remaster of last of us, nor horizon as they want me to buy a console to play it. So what's the point?

Reply on news.ycombinator.com