Restoration of defocused and blurred images (2012)

Of course you can't retrieve destroyed information.

But information is not destroyed by perfectly unpredictable (uniform) distributions of noise.

> Yes, your result would be a very clear image of one possible license plate.

Fortunately there are methods of evaluating how well your statistical filter works, if it's precise enough you'd be fine with indeterministic nature of your filter. Or even better, you could generate all of the highly probable licence plates, instead of having only one - given by your deterministic algorithm.

Not really. Upscaling is hardware, not some ridiculously slow software solution.

https://community.sony.co.uk/t5/blog-news-from-sony/inside-4...

Upscaling is a very state of the art technology, not some layman's solution.

There are firms that specifically targeted upscaling as their product and made millions with their state-of-the-art tech. Currently upscaling is in the rise again with 4K TVs. Back in the days they made some incredible chip solutions, sold them expensively to Sony, Samsung and similar. Sony realized they can, with all of their resources (super-HD movie database) make incredible upscalers.

Just imagine that Sony has stored whole movie The Walk (distributed by Sony Pictures) in your TV in 4K resolution, the moment this movie is displayed on your screen through some lower resolution sources, they find it in the database and display the 4K content. Of course, that's highly inneficient and memory intensive, thus, they use statistical models to efficiently store movie material and have fast chips to quickly approximate the real upscale.

This will then, if the sample (number of movies) is high enough, work well on all of the movie content.

In focus-blurred images, the blur spreads the information about certain pixels throughout a region of the image. (Objects appear larger and fainter because of this when they are blurred. It's why you can shoot through a mesh-wire fence with a wide lens aperture, for example, and not see the fence.) This 'spread out' information is being used to reconstruct an approximation of the original image.

However, when images are downscaled, all the information from 'subpixels' is kept within the reduced-resolution-image-pixel, and all replaced with a single RGB value. There is no 'spread out' information from those pixels. The 'blur' only applies to a region with sharp edges that exactly coincide with the unit of information. So, none of the data from the sub-pixels remain. To reconstruct the pixels that were within you have to essentially guess based on the context of the surrounding pixels.

If the information that belongs in one pixel is spread out over 50x50 pixels, according a known mathematical formula, you can reconstruct it pretty well.

If you just have one pixel with the average of what it covers, then that is all you have.

You can! The problem in that case is it's not a convolution (blur) per se. Each pixel is an average of subpixels P(1)=S1(1)+S2(1)+...+Sn(1), P(2)=S1(2)+...+Sn(2) -- but as you can see there are no elements in common between pixels, which is the case with convolution. That is, in the deconvolution case there are as many variables as unknowns, whereas in the upscaling case you're creating unknowns out of a single variable.

To estimate those subpixels then you're going to be forced to make additional assumptions; if you assume they are independent you would simply estimate Sk(j)=P(j). The traditional (easiest) assumption is that the image is somewhat "bandlimited" -- it does not have many variations (frequencies) faster than once per pixel (so no equal or faster than subpixel variation). If this were the case, you could reconstruct the subpixels perfectly [1], save for some noise. But this is not always the case (and fails spectacularly when you have edges), resulting in upscale blur. So simple linear upscaling algorithms by reaching a compromise between blur and edge enhancing.

If you want to do better though, you have to use non-linear kernels and have good underlying models for your image content. A promising approach is to use machine learning/NNs: http://engineering.flipboard.com/2015/05/scaling-convnets/

[1] https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampli...

"Subpixels" are an artifact of the display type you're using. Although there may be individual colour components of an image pixel, they aren't restricted to a region of that pixel. And in most digital images - those made with a colour filter array, as opposed to a multi-chip, multi-shot, or Foveon-type sensor - the colours at each individual pixel are just an educated guess based on the surrounding pixel values; the RGB values are not captured separately, and attempting to increase resolution depending on which part of your screen is lit up by how much in order to render the colour that should appear at the corresponding pixel can only introduce artifacts. (If green is brighter than red in a particular pixel, it only means that green is brighter than red, not that the detail is laterally displaced.)

I have tried the referenced implementation (SmartDeblur) on a few (similar) slightly de-focused images. From memory, I had the artefacts without any noticeable improvement. I don't think I even saved the results. YMMV and I would almost certainly try it on photographs with a larger blur than mine (approximately a disc kernel with radius of 0.8px). Since then I've been working on other methods with mixed results.

I've found a few things out: Firstly, many articles mention some form of Richardson Lucy with regularization as if de-convolution with a known Point Spread Function is a solved problem. The regularization can indeed produce better results, but usually introduces new parameters. I've found, with real photographs, I have had far better results using less statistically sound methods. This is especially true as I've found that RL, more often than not, begins to diverge quite early.

The reality is that there are papers surrounding deconvolution and blind deconvolution that get apparently good results and work poorly in practice. It is only once implemented that the flaws of the algorithms proposed are exposed. This is compounded by the fact that we have a wonderful error metric with this inverse problem: the euclidean distance between 1. the solution convolved with the point-spread function used for deconvolution; and 2. the original (blurred) image. With many algorithms, lowering the euclidean distance decreases the perceived quality by introducing more artefacts. Nonetheless, I have only seen a handful of papers that include this metric when considering a proposed regularisation scheme, and this is one of the only times when this particular metric makes any sense in computer vision -- if your deconvolution is working, and your PSF is correct (which it is in many cases as the blur is synthetic) then the euclidean distance is the measure you want.

I have since switched to least squares using L-BFGS-B for fitting which has lowered again the Euclidean distance and produced the best, most natural-looking results by far on real photographic images. Unfortunately, estimating the PSF is difficult for small out-of-focus blurs. Optics applied to an ideal lens would suggest a disc kernel, but at this scale other factors are playing a part.

I'll probably do a writeup of this when I'm done -- though I don't know when that will be. I'll also be putting any code on Github in due course. In the meantime if anybody has any pointers or is interested in further details or source code please grab my email address from my profile.

I agree that the staircasing effect is definitely the biggest drawback of Total Variation. In the "Smoothed" picture the noise is removed but the results are blocky.

The first way to deal with it is to take into account higher powers of the differences, e.g. using a linear combination p-norms or a Huber function.

The second way is to take into account second order differences. This promotes piecewise affine instead of piecewise constant functions. You can go further and look at third order differences, but the improvement is minimal.

Other than being more complex, the biggest downside is that all of these methods have some new parameter(s) to tune.

Woah, lesson learned. Thanks for the link.