Microsoft Unveils Its Own Version of Nvidia's RTX Super Resolution | Hacker News Reader

Microsoft Unveils Its Own Version of Nvidia's RTX Super Resolution

tomshardware.com

78 pointsFinnKuhn71 comments

I wonder if we will see something like this in MS Teams at some point as a lot of my colleagues seem to sit in front of 240p webcams from the 90s with the most atrocious lighting you can imagine.

I'm afraid there's little any algorithm can do if the video you feed it with is that garbage quality, unless it just rebuilds it from scratch using pose estimation etc. but at this point it's just a brand new video.

>there's little any algorithm can do if the video you feed it with is that garbage quality

Have you seen the RTX demo vids? It does seem to defy garbage in garbage out somehow...

When you artificially upscale faces it’s easy to see when it guessed wrong what your face looks like.

I could see live video chat benefiting from you taking a couple close up pictures at the beginning of each chat. Imagine face unlock taking images of your face to later use to deepfake you onto yourself.

ML can do much better than that. It can look at noisy, low resolution video and reconstruct what it would look like on a good camera. Not really an algorithm in the traditional sense.

I intentionally use a playstation eye webcam...

A lot of the resolution issues aren't the camera.

The providers (zoom, webex, MS) have lowered the resolution to lower their bandwidth costs.

> The providers (zoom, webex, MS) have lowered the resolution to lower their bandwidth costs.

Why? Could they NOT do something similar to it - https://blog.google/products/google-plus/saving-you-bandwidt... ?

more ideas - https://www.infoq.com/news/2020/10/nvidia-video-bandwidth/

Because these laptops often have dreadful compute performance as well.

Honestly, it's 90% of the problem. No doubt the crappy bitrates don't help, but it's mostly the camera. Even a fairly low bitrate 480p stream with a good camera looks great.

I've recently started using an old iPhone as a webcam and the differences are startling. Especially in bad light.

Though, it is frustrating seeing how much the bitrate ruins it. Comparing the 1080p output from the iPhone to what you see on zoom etc is depressing. But the camera alone can vary from abysmal to very good. Can't get excellent because of the bitrates though.

Does anyone know if it will ever be possible to use this technology on demand to upscale local (not streamed) video files?

There are already NN based shaders you can use to upscale video locally such as in madVR and with mpv and FSRCNXX. I'm not sure how that 'AI upscaling' is related to the algorithms used for this.

You can already use 2D networks, and they work pretty well, but there actually is a dearth of usable temporal video upscalers even though there are like 100+ papers on the subject.

You can already play many local files in formats like mp4 directly in Edge or Chrome to get Nvidia VSR working, probably works with the new Edge VSR too.

If you want a media player that does this and have a 30 or 40 series Nvidia card, you can use the below custom renderer being actively developed, which uses Nvidia VSR tech with a custom renderer for Media Player Classic.

https://github.com/emoose/VideoRenderer

People have also been using MadVR to upscale videos which uses a similar renderer but I think it doesn't have GPU limitations.

I should have been more specific and said that I want to upscale videos and then save them, not just view them. Thanks for the suggestions though.

I think that's what Topaz AI does? https://www.topazlabs.com/

I've seen AI upscaled stuff on the pirate seas lately so the tech already exists & is in active use for non-streamed.

I don't know the details unfortunately

https://www.topazlabs.com/topaz-video-ai

And to be more specific, for non-realtime transcoding you want to use ESRGAN in VapourSynth with models from upscale.wiki. You generally want a denoiser in front of the AI model and a temporal softener behind it. This is quite a rabbit hole, and I can point to more specific filter setups if you want.

For realtime, mpv shaders and mpc-be extensions linked above are good. TBH I havent done a subjective quality shootout between these in awhile.

Smart TVs also have AI ASICs that can do a surprisingly good job. That + vapoursynth transcodes is what I lazily rely on these days.

I'd be interested in more info about the non-realtime upscaling. I've used ESRGAN before, but never a temporal softener. Can you share some best practices?

Ok, so on the one hand I love the amount of features and innovations Microsoft keeps bringing to Windows and the likes of Bing Search, but on the other hand Windows and Edge are now just adware.

Just look at the screenshot picture in the article. One third of Edge's windows is just ads for games.

Remember the days of IE 6-7 when you'd get them infested with toolbars and adware as every Windows app you'd install would try to sneak those in during the installation process if you weren't careful to untick the right boxes? I guess someone at Microsoft though "hey, what if now we were the ones pushing the adware to our customers and monetizing their data?".

Nailed it. Microsoft forced me to Google through bugs. Google forced me to Apple through advertising and privacy violations. Microsoft and Google are now doing very well at keeping me in Apple's ecosystem.

I periodically use an android device or sample windows just to see if it got any better and, no, it's a veritable shit show. They had good, mature products and screwed them up by selling out as adware companies.

Android at least used to be a relatively open platform where you could own the system but of course Google screwed it up by adding remote attestation. Now it's either iPhone or postmarketOS.

Don't iPhones also employ remote attestation?

They do. What I meant to say was now that Android is locked down there is no point in accepting its shoddiness anymore. If I must be in a walled garden, I'm choosing the best kept garden.

Ah, I understand now. For me, iPhone doesn't have table-stakes until I can install apps off GitHub. Maybe come October with the regulation and a new USB-C iPhone, my mind will change.

I really didn't want to switch. I don't know how to live without Termux anymore. Google's screwing up even that, now vital system calls like execve are no longer usable by apps so Termux won't even be able to spawn programs anymore. They worked around that by moving to F-Droid but who knows what Google is gonna screw up in the future. What's the point?

My plan is to buy an iPhone soon and then try to port postmarketOS to my current phone.

> What's the point?

Knowing that when things go to shit, you can at least have a cell phone?

I get your fatalist approach here, I really do. But if you think authoritarian companies are going to eventually ruin everything, I don't see why you'd go all-in on Apple. If you feel shorted by the Play Store's removal process, you won't like the App Store any better. If you're mad about first-world features like hardware attestation existing and being used, that's less Android's fault and more the modern development culture moving forwards.

Do whatever makes you happy in the end. Long-term though, I cannot imagine myself switching back to iOS. Apple has nothing like the AOSP, which love-it-or-lump-it is the best thing Android has going for it today.

>Apple has nothing like the AOSP, which love-it-or-lump-it is the best thing Android has going for it today.

ELI5, what's AOSP? Is it like, base android without hardware drivers? Does this make it easier to maintain 3rd party android roms?

> Is it like, base android without hardware drivers? Does this make it easier to maintain 3rd party android roms?

More or less. The AOSP is the free portion of the Android codebase (licensed in Apache) and actually accounts for the vast majority of Android code you run. Google then adds their services to the OS and preinstalls it on most phones. Most people derive custom roms from a combination of AOSP code and official (albeit proprietary) drivers.

I can still own my system just fine by choosing a phone with good community support and installing LineageOS (or some other variant of Android) + rooting it. If you don't want Gapps but still need to run apps which rely on it either microG or GrapheneOS with its sandbox are potential options. SafetyNet can also be bypassed if needed.

Now sure, that requires some extra effort and time from the user but it's just the situation we are in currently. Not like you have any other current options for an open mobile system outside of being an early adopter of GNU/Linux phones and their ecosystems.

> I can still own my system just fine

> SafetyNet can also be bypassed if needed

Nope. Android phones now have hardware attestation, good luck faking that. If you install any of the stuff you mentioned, you've "tampered" with your phone. Hardware remote attestation will now fail and apps will no longer run.

My bank app complains about developer mode for god's sake. If WhatsApp starts requiring attestation, I will have the "choice" to either give up my freedom or enjoy my $600 paperweight.

Had no idea that was a thing. Thankfully my current phone appears to be old enough to not make use of it. That sucks, can't wait for the mobile GNU/Linux ecosystem to mature.

It was a shock to me when someone here told me about it. Android is finished. My plan is to buy an iPhone and try to port postmarketOS to my current phone instead of insisting on Android.

Sorry you are denied a workflow, but I hope I never need to install a banking app on my phone. Smartphone is now such a liability for the power it has over my life. Unreasonably nervous about what happens if it breaks/lost/stolen.

I don't get it. Two advertisers are keeping you in the ecosystem of.. a third advertiser?

I could see a Linux or a BSD, but Apple is also just another advertiser spying on you here.

I don't get ads on iOS, macOS or iPadOS. Literally none.

The only revenue stream they have in that space is in-app ads (I don't have any of those apps) and the app store, which is basically an advertising portal to find tools so I expect it there.

Google and Microsoft shovel it down you and bug you constantly about upsells and third party trash.

Every time I search on my wife's iPhone for some specific application in the App Store the first result is an add for a competitor's app. How is that 'expected'?

How is that not expected, let alone “spying” in anything like the way FB or Google do?

I don’t like ads in general, but “you searched for a competing product, please consider mine” is so old school and understandable.

I use my desktop and perform searches every day. I am constantly nagged with targeted ads and bulk promotions when doing so on Windows and when using Google.

I search for apps in the App Store once every few months. Perhaps others do it once or twice a week if they like exploring and trying new apps.

Does that not seem qualitatively different to you? Even if Apple makes some revenue as a result of advertising inside their fully segregated marketplaces, it's hard to think that those ads are a big part of one's experience with Apple.

Yeah I don't get any either. I also don't think they're selling your data (or if they are, they're very bad at it) because my targeted ads don't reflect my activity on their OS...

Unlike my android phone. My youtube ads change noticeably in response to whatever I've recently googled, it's very unsettling how dialed in they are.

My daily driver is Linux though. I do agree that's what you should be using if you want to be free of these dystopian shenanigans.

https://www.bloomberg.com/news/newsletters/2022-08-14/apple-...

"Let’s begin with the current state of play: Apple’s advertising efforts today consist of display ads inside of its News and Stocks apps, as well as inside the App Store, across the iPhone, iPad and Mac. The App Store also has Google-like search ads. And more recently, Apple put advertising inside of TV+ for its “Friday Night Baseball” deal with Major League Baseball."

https://www.forbes.com/sites/daviddoty/2023/02/08/apple-the-... has more on the near term changes you can expect. Addendum: https://www.vox.com/recode/2022/12/22/23513061/apple-iphone-...

These are the adverts I want: discovery in scope of what I am doing i.e music, entertainment and apps. Literally Apple deliver those cleanly.

What I don't want is scams, noise, garbage and malware which is what the alternative ecosystems are full of.

That is literally the "Games" pane you can turn on in the settings. It's just the store for their games. It's not the default Edge experience. You only see that when you purposefully turn it on.

Nvidia recently rolled out upscaling in video too with new drivers and latest Chrome/Edge [1]. Honestly, it's pretty decent from my usage. Required a 3080 or above, however.

1. https://arstechnica.com/gadgets/2023/03/nvidias-new-ai-upsca...

I wonder if it will be possible to use this to convert video files using their super resolution, and not just web pages playing videos

I've tried the nvidia one and it's really not particularly noticeable. On low resolution media (eg below 720) it really looks like it has removed a lot of the details - mostly as at such a small resolution, there aren't any details haha, but up scaling makes it very noticeable.

There are much better options for non-realtime.

just saw your comment, any recommendations for non-realtime video upscaling?

It doesn't work on Intel's GPUs?

It supports Nvidia RTX 20/30/40 and AMD RX5700-RX7800 series GPUs

That's not surprising. Intel's best GPU is nowhere near as capable as the minimum GPU Microsoft is supporting here. The NVidia 20xx series is still pretty beefy.

Even so, the reason for the support is most likely because the supported range includes GPUs with compute units (CUDA cores etc), which I think Intel doesn't have?

Discrete graphics cards have had CUDA cores for ages. Maybe this tech uses Tensor cores, that's why the support starts at the 2000 series of Nvidia cards when Tensor cores were first introduced. Not sure on the AMD side.

Yeah, of course, I wrote CUDA when I meant Tensor/RT cores. Thanks

AMD 6000/5000 series have no matrix acceleration at all.

7000 series matrix ops are built into the shaders, like Intel Arc (as opposed to being a seperate block on the GPU like Nvidia). Also the instructions are different than the datacenter CDNA line (while the instructions for Nvidia datacenter/consumer cards are the same).

The A770 is more powerful than a 2060. 20 series is like the minimum because it makes use of Tensor cores which the 1660 and below lack which is why those can't do DLSS either. The Intel Arc cards can do similar tricks with XeSS and image/video upscaling is possible on Intel Arc GPUs using software like the stuff from Topaz Labs.

Arc's FP16 matrix throughput is actually quite good, way better than the AMD 6000 series.

Looks like it is in the works:

https://videocardz.com/newz/intel-is-also-preparing-video-su...

I think the Microsoft's one may have been based on Clarity Boost[1] rather than Chromium's VpSuperResolution since there's still no AMD implementation in Chromium[0], but Edge VSR does support AMD cards.

A fun anecdote here is that VpSuperResolution was actually added to Chromium by Intel back in April 2022[2], citing Clarity Boost as an inspiration. It works with iGPU, but it was never enabled by default. Nvidia's patch came much later in Sep 2022[3] but enabled by default with a driver-level trigger (rather than enable behind a feature flag like Intel).

[0]: https://chromium.googlesource.com/chromium/src/+/master/ui/g...

[1]: https://www.microsoft.com/en-gb/edge/features/clarity-boost?...

[2]: https://bugs.chromium.org/p/chromium/issues/detail?id=131838...

[3]: https://bugs.chromium.org/p/chromium/issues/detail?id=136831...

Might also be interesting then to use Edge as a video player for local lower-quality video files. At least for owners of AMD GPUs, which currently don't provide this feature through the driver. Does this work in the Linux version of Edge too?

So is this is something that can be incorporated into the next generation of video encoders? Will h.268 or whatever require decompressors to come with a 1GB pretrained model?

I wonder how small you could get video with an on disk model and texture pack. Diff video blocks from samples in the texture pack, then reverse discover what input the model needs to recreate the diff.

Future videos will just be text prompts used to generate videos in real time.

Back to channel flipping. Endless procedurally generated channels that continue forever and never stop existing. Watch it live or watch it from any point in the past at 2x

For what it's worth, Chrome already supports RTX Super Resolution on videos and an early review[0] seems to indicate that it's hardly an improvement.

So while I do think this is nice in principle, it's not really exciting yet and it's far from being a reason to consider using Edge.

[0]: https://www.techpowerup.com/305296/nvidia-rtx-video-super-re...

Reply on news.ycombinator.com