Photoshop 'unblur' leaves MAX audience gasping for air
9to5mac.com
9to5mac.com
(slides: http://cs.nyu.edu/~fergus/presentations/fergus_deblurring.pd... (~60 MB ppt) paper: http://cs.nyu.edu/~fergus/papers/deblur_fergus.pdf (~10 MB pdf) )
The basic idea is that you have an unknown original image and it is convolved with an unknown blurring kernel to produce the observed image. It turns out that problem is ill-posed. You could have a bizzare original image blurred with just the right bizzare blurring kernel to produce the observed image. So to estimate both the original image and the kernel, you have to minimize the reconstruction error with respect to the observed image, while penalizing unlikely blurring kernels or original images. If one extracts enough statistics from a dataset of natural images, one can tell whether an image is likely or not by comparing that image's statistics to the corresponding statistics of your dataset of natural images. Similarly, simple blurring kernels are favored over complex ones (think "short arc of motion" vs. "tracing the letters of a word with your camera")
In one dimension, you'll be looking at a slice (more accurately, a summed projection) of the kernel. Focus = guassian, motion = square wavelet (line) or irregular (arc)
Here are the actual research papers for this video:
http://people.csail.mit.edu/sparis/#publi2011
- Blur kernel estimation using the Radon Transform http://people.csail.mit.edu/sparis/publi/2011/cvpr_radon/Cho...
- Modeling and Removing Spatially-Varying Optical Blur http://people.csail.mit.edu/sparis/publi/2011/iccp_blur/Kee_...
----------------------------------------------------
http://knuthlab.rit.albany.edu/papers/knuth-ica99.pdf
Abstract: The problem of source separation is by its very nature an inductive inference problem. There is not enough information to deduce the solution, so one must use any available information to infer the most probable solution. We demonstrate that source separation problems are well-suited for the Bayesian approach which provides a natural and logically consistent method by which one can incorporate prior knowledge to estimate the most probable solution given that knowledge. We derive the Bell-Sejnowski ICA algorithm from first principles, i.e. Bayes' Theorem and demonstrate how the Bayesian methodology makes explicit the underlying assumptions. We then further demonstrate the power of the Bayesian approach by deriving two separation algorithms that incorporate additional prior information. One algorithm separates signals that are known a priori to be decorrelated and the other utilizes information about the signal propagation through the medium from the sources to the detectors.
http://knuthlab.rit.albany.edu/papers/knuth-eusipco05-final....
Abstract: Source separation problems are ubiquitous in the physical sciences; any situation where signals are superimposed calls for source separation to estimate the original signals. In this tutorial I will discuss the Bayesian approach to the source separation problem. This approach has a specific advantage in that it requires the designer to explicitly describe the signal model in addition to any other information or assumptions that go into the problem description. This leads naturally to the idea of informed source separation, where the algorithm design incorporates relevant information about the specific problem. This approach promises to enable researchers to design their own high-quality algorithms that are specifically tailored to the problem at hand.
It's nothing new really, but algorithms for it have advanced tremendously. For example, there's some results from 2009 http://www.youtube.com/watch?v=uqMW3OleLM4
Teuobk on HN also made a startup/app based on this, but it seems to be down now: http://news.ycombinator.com/item?id=2460887
Probably would take a massive cloud of systems to correct any highdef video signal, but it would be impressive for many applications (news broadcast, sports, or any live event, security, or remote robots)
Granted to achieve performance on the order of near realtime dsp, it would require an impressive hardware system. Then again, when I can spend the price of a coffee and get access to a cloud of cpu's...
As I understand it, their system is trying to estimate the causes of noise in the image (camera shake, badly focused or dirty lens, etc) and apply an inverse transformation to produce the original 'real' image.
They can't extract detail out of nowhere, so a lot of it will likely be heuristically driven, accounting for specific sources of error and applying adaptive transforms with some sort of 'looks like what we think it should look like' cost function.
Figuring out the cost function is probably one of the hardest parts here.
In terms of video, you've got more data to work with, which in theory means more constraints to your solution making it potentially more accurate, but I suspect it'd also be very easy to get bogged down in the quantity of data.
Combining and merging data from multiple independent sources is a different problem entirely, probably more like synthetic aperture imaging: http://vision.ucsd.edu/kriegman-grp/research/synthetic_ap_tr...
But yes, most image processing techniques like this lend themselves to video. This is something I hope my startup will someday be able to integrate into our collaborative video editor...
http://camcorders.about.com/od/camcorders101/a/optical_vs_di...
In theory, you can tweak this method such that it is run in parallel frames 'almost' independently. I say 'almost' since you will have to factor in temporal coherence, such that deconvolution kernels do not vary much between subsequent frames (otherwise it will cause unwarranted flicker and jitter like effects)
However, more pertinent and related problem with respect to video is that of image stabilization. And computer vision community is making some exciting strides in the front; for eg: http://web.cecs.pdx.edu/~fliu/ .
On a side note: does anybody know of workable deconvolution algorithms that vary the kernel over the image? The example would be compensating for a bad lens.
(http://adsabs.harvard.edu/full/1992ASPC...25..226K)
What an awful URL.
Sorry for OT, was just happy to see the coincidental link to a Hubble photograph.
Adobe's noise processing algorithms in their software are something else, especially in ACR6/Lightroom 3.
I moved 'down' from a 5D2 to a Panasonic GF1 as I don't shoot pro anymore, and ISO1600 on this thing with a slightly-off exposure can be pretty noisy. Lightroom cleans it up incredibly well without losing clarity/sharpness. Before that, I'd just do all I could to keep the ISO down so I didn't 'lose' images to noise.
The basic theory for doing this type of deblurring isn't too bad, but making it easy and automatic becomes a really difficult computer vision-related problem. Adobe has been working on this (and a lot of other pretty cool stuff) for quite a while. It'll be interesting if they ever ship it.
For example, if you see what looks like blurry skin, and you de-blur it into skin, no-one will complain. Unless your de-blur noise elimination thinks the blurry eyes are noise, and turns the subject into a faceless monster.
"OH, oh I'm sorry, but this is abuse."
"Oh, I see, well, that explains it."
"Ah yes, you want room 12A, Just along the corridor."
Honestly though, if people were enjoying them, I don't think it's any less rude. It says more to me about the anti-intellectualism mentioned earlier. Reminds me of the Diesel campaign surrounding the word "Stupid".
I don't know, I think you can nail this kind of repartee and it's funny and makes things flow - which stops the day from getting boring and stiff. But I hate it when it's not quite right. Kinda like the MC at a gig who tries to make bad jokes and ends up being super awkward.
It was a pretty funny way of asking for more details imho.
That was a little too hand-wavy. I'm a little dubious until I see what went into that phase.
The parameter preload isn't cheating -- if they're anything like the InFocus params, they're pretty obvious but somewhat tedious. They're things like telling it that you're trying to correct motion blur rather than focus blur, what level of artifacting you're willing to put up with (for forensics or text recovery, you can put up with a lot of noise in the uninteresting part of the picture), the desired hardness of recovered edges, that sort of thing. It would have just been a time-waster for the demo (and, like in the demo, InFocus allows you to save the params as a preset).
I suspect people will have to manage their expectations with this Adobe plugin/feature as well.
I sort of expect Adobe to do better -- they've got a lot more resources to work on the problem. Maybe I'll finally find a reason to upgrade from CS3.
When I tried InFocus, I used them on some shots where I blew the focus at wider apertures (now that I'm using a camera with better high ISO performance, that kind of defocus is rarely an issue, because I have the luxury of stopping down), and I couldn't get adequate results and I wasn't willing to spend a lot of time tweaking sliders.
I am totally with you on the idea of taking better pictures though. The more you can reduce your effort in post with technique, the better. A lot of stuff is unfixable in post.
How does it disappear at the end - or is that a virtual digital overlay?
Wait, is the entire background rear projected, like a borderless movie theater screen? Must be massive resolution ?!
As for "massive resolution", slicing up a framebuffer and shooting out the components to multiple projectors wouldn't be a new idea, and I'll bet that's what was done here.
This is seriously cool technology.
"zoom in on that. good. now....ENHANCE."
Upscaling an intrinsically low-resolution image is still in the realm of creating information where there was none, I think.
what CSI does it add information that wasn't initially there whereas what this is doing is just unscrambling the information. This is only for photos where the camera has moved during exposure, so should be great for low light shots (indoors etc) where you need a shutter speed a bit slower than optimal, say 1/5 second
The Adobe approach, on the other hand, handles complex (non-linear) motion blur and does so in a so-called "blind" way.
[1] http://en.wikipedia.org/wiki/Point_spread_function [2] http://www.optinav.com/Iterative-Deconvolution.htm
The technique to unsteady hands artifacts often has access to motion data, which this thing does not.
Is the 'decoding' paper this one: http://dheera.net/projects/blur.php ? The blur function is just smearing pixel values across their neighbours in blocks, so you can treat it as a hashing function, and then generate enough candidates that eventually you get something that hashes correct (or close enough)
I'm not sure how practical it'd be on data longer than a credit-card number, but it's an interesting hack nevertheless.
http://www.fmwconcepts.com/imagemagick/fourier_transforms/fo...)
This is a perfect "web service" where you can profit from selling the device to have free access to the service.
It might be more efficient to just teach people how to use their cameras so that they can minimize unwanted blurring.
1) Load image. 2) Filter -> Gaussian Blur 3) Undo
;-)
I'm not claiming the demo is fake, I'm just wondering why the video looks so strange.
Super Troopers: http://www.youtube.com/watch?v=KiqkclCJsZs
Red Dwarf: http://www.youtube.com/watch?v=KUFkb0d1kbU