Nvidia Can Artificially Create Slow Motion That Is Better Than a 300K FPS Camera | Hacker News Reader

Nvidia Can Artificially Create Slow Motion That Is Better Than a 300K FPS Camera

motherboard.vice.com

32 pointsjedberg17 comments

Title is misleading. Nvidia have created a network which can slow down (i.e. interpolate) videos by a factor of four.

Actual link: https://news.developer.nvidia.com/transforming-standard-vide...

Most importantly, the footage shot in the video was at 2500 fps native [1] - at 300k FPS you have to use a significantly reduced image size. FPS is bottlenecked by the time it takes to read out the sensor, which is linear in the number of rows in the image. So if you read out half the sensor, you can double your frame rate. You can exploit this to get thousands of frames per second on a cheap machine vision camera.

The camera framerate is largely irrelevant. While you can slow down footage by four times (neat), that doesn't mean you can get 300k fps footage out of a compact camera. The great benefit of this technique is really that you can convincingly fake super-slow motion without having to sacrifice sensor resolution. I can easily see this as a cool iOS or Android feature.

Maybe if they'd actually used footage that was at the limit of camera technology this would be more impressive (I don't see a technical reason why not?). Why not combine it with super resolution too? Interesting security implications, as cameras with frame rates higher than (I believe) 1M fps are ITAR controlled.

[1] https://www.youtube.com/watch?v=5mZovjRlkWs

> as cameras with frame rates higher than (I believe) 1M fps are ITAR controlled.

I'm curious what's the reason for that. I can understand that extremely high resolution could be used for intelligence purposes/high quality satellite images, but high frame rates?

The UK government has a great website for this, where you can look up restricted things and the reason for the restriction. Search for "frames". Mostly it's because they're used for ballistics and weapons imaging.

They're listed under item 6A003 (broadly, cameras) on the export list which includes:

- Cinema cameras with frame rates more than 13,150fps

- Mechanical/electronic cameras with frame rates more than 1M fps

- Frame readout speeds of more than 125 fps full frame.

The last one is interesting, you can buy machine vision cameras with higher frame rates fairly easily, so perhaps it's only something to worry about if you're traveling.

https://www.ecochecker.trade.gov.uk/spirefox5live/fox/spire/

I've seen some exaggerated titles but this one is something else.

It's definitely good frame interpolation, but not only does the slow-mo already look artificial at a mere 4 or 8 times slower than the original it also does nothing to reduce motion blur, which makes it incapable of replacing a high speed camera in any meaningful way.

I guess they may have meant that it can also improve images that come out of a 300k FPS camera, but that's a somewhat meaningless statement. If anything more FPS makes things easier to interpolate.

Not to mention, frame interpolation isn't anything that inherently requires complex AI. My old Samsung TV had this feature, and it created the irritating "soap opera effect".

If your TV created the soap opera effect and this new method does not (does it?), then the new method is clearly better and maybe it did require this complex AI after all to make that happen.

The soap opera effect comes from high FPS playback vs. the aqcuired taste of the 24fps cinematic framerate which is independent of the topic of slow motion and is technically not a flaw, it's an aesthetic preference.

Actually, I don't think it's inherent to high FPS playback. It's also that the capture is more fine-grained. I imagine you could capture 300fps, such that each frame is captured over 30ms and play that back at 300fps and retain the cinematic feeling that you are used to. I don't know if doing that would actually bring any of the benefits that normal higher fps capture brings.

The soap opera effect is due to very high frame rates breaking the fourth wall and making it clear that the thing you are watching is horribly faked. You can notice that the effect is not nearly so severe on non-staged content (sports, wildlife, news)

Ok, got it. So even the best quality slo-mo interpolation would create tbr soap opera effect.

I should have googled that first. :-)

Some alarming spots - the ice skates on the hockey player, the planted foot on the dancer - but otherwise it looks pretty convincing.

Whats the big difference with this and https://www.svp-team.com/wiki/Main_Page ?

This may appear to "work" but I doubt it is better than a 300K FPS camera by any means. It will not work perfectly across all scenes, it cannot interpolate uncommon movements and finally, it cannot interpolate many low-res cases like 5-20 FPS where there is actually significant missing information across frames.

That interpolates look to me significantly worse in case of highly volatile volume affecting reactions, like the ballon and the the immediate jelly post-impact sequence with the racket. There a clearly still hard limits to what you can believably hallucinate without knowledge of the higher frequency processes that have not representation in the source material and are to chaotic to be learned from watching random 240fps videos for training.

Also for a research output the ground-truth real slomo vs. super slomo is missing to make a proper quality assessment. The paper itself only list some fairly low-frequency inbetween frames as reference for comparison with other methods.

https://arxiv.org/abs/1712.00080

Clickbait. It needs the 300K camera input in order to create the output. edit: misspelling

This is really cool. Seems like a better application of neural networks on video than the deep fake stuff earlier this year.

I can see issues with the blade on the ice skater. Very cool though.

Reply on news.ycombinator.com