Magical, 3D-Warping Techniques Steadies Your Videos | Hacker News Reader

Magical, 3D-Warping Techniques Steadies Your Videos

createdigitalmotion.com

71 pointsteej29 comments

Stabilization into a smaller rectangle discards material around the edges of the original larger video.

So in a sense, this is yet another case of 'more data' (the oversized source video), plus software/CPU batch analysis, replacing the specialized equipment and expertise that used to be required for a 'stable' shot.

Taken to its logical extreme: might future casual-use 'cameras' be compact omnidirectional arrays of high-resolution, almost-always-on photo sensors -- from which idealized photos and videos can be reassembled later, by post-processing and editing at leisure?

Sure. But noise imposes a fundamental limit on resolution.

more sensitive equipment developed in the future would lower the noise floor, to the fundamental limit of resolution.

Will that equipment enable us to know exactly where a photon from the subject has hit the detector?

And the human eye imposes a fundamental limit on how much resolution is needed to faithfully reproduce a scene.

Which limits are closer: the eye's capabilities or the noise floor?

What does that matter? The comment speculates about the ability of an arbitrarily small device to capture an arbitrarily large amount of information in order to crop it to the region of interest later.

The speculation was also about "casual-use 'cameras'".

Your point that theoretical limits are eventually reached is well-taken, but tiny devices could be collecting far more than the unaided eye perceives before hitting physical limits. Eventually, the devices could compete with microscopes and telescopes -- and then the "fundamental limits" you've highlighted will become a issue.

There are situations where you cannot have specialized hardware on set, because you don't have a set - for example image stabilization is a key component in film restoration.

For example: http://www.davsys.com/revival.html

and http://www.hs-art.com/

those two are what I've seen and used in practice and both have A LOT to be desired for.

I always said whoever solves automation of object recognition and pixel perfect outlines of them is a VERY RICH man. Current problem solving revolves around specialized software for rotoscoping masks around objects with a help of 2D and 3D tracking here and there. It is a very tedious job that requires a lot of man power. See, for example, http://www.mokey.com/

Is it just me, or is the warping sort of noticeable?

It feels like there's some sort of other-worldly quality... I think it's parts of the mesh moving around out-of-sync with other, but it's really hard to get a sense as it's one of those things where it's the image as a whole, not any one point you can look at.

Honestly, it makes me slightly nauseous - I'm not sure I could deal with it for a significant period of time.

It reminded me of animated movies where the 3D-scenes are drawn all by hand.

No nausea here. I want to see this technique used in real movies... great opportunity to get something that feels new on the screen.

I've watched it twice now and I only noticed warping during the scene with the water in the background and maybe a smidge on the big road cone in the stairs scene (though that might have just been some exposure shifting).

I figure they've still got some bugs to work out anyhow.

I think a lot of the otherworldly quality is simply that they've put the camera on a line. Not "sort of a line", not "a steadi-cam pretty-close-to-line", but a line. Very few shots ever are actually on a rigid line, especially at a single rigid speed. Even track shots will do some panning, or travel at different speeds throughout the shot.

This is easy to fix, of course. For demonstration purposes, it really drives the point home that they have stabilized this picture, no ifs ands or buts. For real use, I'd probably stick with the lowpass filter. Also, in real use, you'd be able to configure the algorithm selectively; line here, lowpass there, etc.

This reminds me of another thing I've never got round to writing - a simple program that just presents your video as a picture within a larger area, which moves around to compensate for your hand wobble. All the solutions I've seen either crop or warp, trying to hide the wobble, but losing or distorting part of the image. (Admittedly I haven't checked this one out yet, because I'm at work. Maybe there really is no distortion). I suspect that the simple solution would be just as good, if you didn't mind letting people know that your hand wobbled, which most people don't.

They make this sound like new but this 3D tracking technology has been available for years & used on feature films all the time for stabilizing & placing CG elements in to live-action film. These are a few software companies that sell this technology & some of the vfx companies have their own in-house developed software:

Syntheyes- http://www.ssontech.com/

Boujou 3D- http://www.2d3.com/

PFHoe- http://www.pfhoe.com/

I think you missed the point, their key innovation is to use a content preserving warp on a frame-by-frame basis to recreate almost ANY motion path they like ( provided that the path is close enough to the original path of course ).

The software you listed are simply a part of the system which they developed use, the tracking/scene analysis part of it. And you are right, that part of the software is not overly difficult to write, in fact you can throw together a Harris Corner detector + RANSAC to find correlation between the points and it'll do quite ok. In their case, they decided to use the Voodoo camera tracking system available here:

http://www.digilab.uni-hannover.de/docs/manual.html

which is comparable to the software you have listed

What I'm most interested in is the speed of processing their warps. From my experience of playing with ITK/VTK, generating non-rigid warps using b-spline, MFFD or any of the current algorithms takes a LONG time if you take the image content into account. I need to read the paper more carefully to find out.

Quite so, but those solutions are very expensive either in terms of up-front costs or labor to apply. This has a 'just works automagically' aspect that's going to make it very popular, enough that it might well show in camera firmware within a few years if it can be sufficiently optimized.

Although this will be greatest interest to indie film and TV makers right now, selling to that market is a loss leader for most companies - the big bucks come from hobbyist consumers that want to shoot pictures or video but are happy to get an assist from the camera - hence the obsession with good autofocus on most camcorders, whereas pros are willing to pay a lot more for manual control of the focus plane. This is instant Win for whoever implements it first - it'll make consumers feel like pros, and it will save pros LOTS of money.

This looks like novel approach and a good one. It is worth pointing out that it looks like it will only work for tracking shots and not static shots that have some shake. I'm not sure if it will work for nodal pans or not either.

Either way, this could be a nice tool for independent filmmakers who don't have access to Steadicam gear. Big budget films don't rely on post processing to fix this sort of thing unless they have no other choice becasue of the constraints of the shot.

The writer only mention's Adobe's involvement in passing. Personally, I think it's a lot better than Final Cut's motion smoothing filter (although the latter has the advantage of being available now). Bad news for Apple. It's a subjective impression, but I feel that FCP is steadily losing ground to adobe in this market.

It's an odd effect that the output, despite playing in real time, has a slow-motion quality to it.

I'm always amazed by this stuff, but I wish more of it turned into available implementations...

Me too, considering how shaky my 5D2 video usually comes out. I'd be happy with a cheap, handheld stabilization system and some decent software. Any recommendations?

Jonny Lee, the brain behind many of the wii-mote hacks, has a $14 "Poor Man's Steadicam" that he came up with:

http://steadycam.org/

Both for windows

Virtual Dub http://www.guthspot.se/video/deshaker.htm

AviSynth http://avisynth.org.ru/mvtools/mvtools2.html

Graphics developments such as these (and many others presented at SIGGRAPH) do indeed turn into real software quite frequently, sometimes as part of upgrades to existing softwares. Given that this work was done partially by people at Adobe, there's a good chance they will include it in the next version of their video-editing tool.

In general, the computer graphics industry is incredibly fast at picking up research developments (often under a year for many of the CGI/FX studios). Other areas of computer science typically have much longer lags between research results and industry adoption.

I think the key of this algorithm is that it utilizes the standard SFM algorithm. If structure from motion is so mature, I can expect the 3D movie without 3D camera in very nearly future.

Nice, that tech is getting close to render steadicams obsolete.

I doubt it will for professionals, at least in lots of shots, the distortions would be considered fairly unacceptable in most movies. Those shots are dominated by static nicely edged structures, with the dynamic parts making up only a small part of the frame.

Still going to be a great tool and looks like a small enough iteration on current stabilisation that it will actually be implemented.

It's quite reasonable to assume that with further engineering and sufficient computer power, many of those distortions will be ironed out. The cost of digital post-processing is in constant decline, whereas the cost of shooting with a steadicam/ dolly/ crane on set is more or less constant. All of these require expensive, bulky, and highly visible equipment for one thing; for another, any time you take the camera off a tripod the cost goes up because moving the camera adds 3 new vector variables to the shot calculus, requiring more takes for a good result.

I see this being an instant hit with low-budget shooters who are prepared to lavish time on getting it right and are more concerned with energy than perfect fidelity. Later there will probably be rough-quality realtime preview on set followed by near-perfect post rendering.

Reply on news.ycombinator.com