Every time someone brings up Super-resolution, I like to pull up this hilarious example : https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a...
Super-resolution is only guessing. It's ok for art, not for critical tasks.
Super-resolution is only guessing. It's ok for art, not for critical tasks.
Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems. Some modern photographic systems actually do this directly (piezo motors?) on the sensor.
Of course this only works if what you are imaging is reasonably static over the time needed to take all the images.
You can do an approximate version of this with video, with caveats because you don't control the motion. The key thing is, though, you actually have more data to work with.
This idea ran in parallel with image processing people attempting to estimate higher resolution from a single image for a while, and unfortunately the terminology stuck in image processing also. Something like resolution extrapolation is probably better but that ship sailed ages ago.
User 'twic' posted a link to a very interesting article that describes this and also explains the difference between photosites and pixels:
https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...
Also, cameras have had stabilization systems for a while now; I would assume they need similar pixel-scale precision. Some cameras shift the lens, some shift the sensor, but either way they need to shift the image-on-sensor by a very small amount, and also do it very rapidly.
[1] For example: https://www.pro-lite.co.uk/File/psj_piezoelectric_nanopositi...
I think in general satellite imaging is a good place to look for such implementations, since they have a naturally and predictably moving imaging system.
https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...
I also have an Olympus E-M1 MkII, but I haven't tried the high resolution mode yet. You just gave me a TODO item!
https://en.wikipedia.org/wiki/Image_stabilization#Sensor-shi...
https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...
https://www.nikonimgsupport.com/na/NSG_article?articleNo=000...
https://www.canon.co.uk/pro/stories/8-stops-image-stabilizat...
There is called stacking, and the subpixel offset comes naturally from path distortion in the atmosphere itself.
I remember using registax when the first batch of consumer telezoom point and click camera came out and got a 30x long exposure of the moon, with, well, mediocre results.
Theoretically, with 30FPS cameras like Sony A1 and said gyroscope data, you can create super resolution images.
IIRC Olympus' some handheld super resolution modes use both shake and sensor shift to increase resolution.
https://ai.googleblog.com/2018/10/see-better-and-further-wit...
> In the early 2000s, Farsiu et al. [2006] and Gotoh and Okutomi [2004]formulated superresolution from arbitrary motion as an optimization problem that would be infeasible for interactive rates. Ben-Ezraet al. [2005] created a jitter camera prototype to do super-resolution using controlled subpixel detector shifts. This and other works inspired some commercial cameras (e.g.,Sony A6000,Pentax FF K1,Olympus OM-D E-M1orPanasonic Lumix DC-G9) to adopt multi-frame techniques, using controlled pixel shifting of the physical sensor. However, these approaches require the use of a tripod or a static scene.
Super-resolution was already the name for this field in this 1987 review! https://kmh-lanl.hansonhub.com/publications/imrecov.pdf But 1987 is pretty recent as far as this field is considered. A. W. Lohmann and D. P. Paris talked about super-resolution this way in 1964! https://www.osapublishing.org/ao/abstract.cfm?uri=ao-3-9-103...
So no. It's not annoying that super-resolution is the name of this field. The name predates the entire field of image processing, it predates the invention of the digital camera! And it is the original use of the title "super resolution", people that do this in hardware adopted the name later.
Because of this, conversations around it are often confused, and it would be clearer if there were a different terminology for the latter.
Isn't this also how our eyes work?
[1] http://imajeenyus.com/electronics/20120908_improving_adc_res...
[2] https://accidentalscientist.com/2014/12/why-movies-look-weir...
Note that 24 fps film cinema runs at 72 Hz flicker due to triple-exposing each frame. That's why they could get away with 24 fps: Since they are dealing with stored images, multiple exposures were possible to avoid flickering. Using as low fps as possible was desirable because film costs money, film transports get more finicky as fps are increased and exposure gets tricky (doubling or even tripling the frame rate leaves you with a much shorter exposure time, which you need to compensate either with a larger aperture (which may not be available, or if it is, cost a lot of money), or more sensitive film (which may not have been available, or has much worse quality), or by having more light in the scene -- all of these are either undesirable or expensive).
So they chose 24 fps because the motion perception isn't quite in the shitter at that fps, though it's pretty bad (nowadays idolized as "cinematic look"; even more hip and cool if you do it on Youtube and deliver 24p which gets converted to 60 fps by means of 3:2 alternation, resulting in a super janky look known as "That Pro Youtuber Look"). This is the reason why people did not like The Hobbit.
Meanwhile video people went on to invent interlacing, because video was fundamentally about a non-stored image. Here the reason why low fps are desirable are again cost, since higher fps needs higher video bandwidth, and that makes everything more expensive and lets you have fewer channels (likely not a concern in the early days). Interlacing lets you have a higher frame rate (a CRT at 25 Hz or even 30 Hz will trigger nausea in everyone) without incurring higher costs, just like multiple exposures in a cinema projector.
“Wouldn’t it have been easier to just look them up in the phone book?”
Pure genius
https://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResoluti...
I worry this is going to be a case where the marketing is at direct odds with public education efforts.
They're still useful tools to have in your toolbox as a photographer or designer, even for critical tasks, and I don't really see how this is different. There may be certain failure cases, but everything has failure cases.
It’s like how modern word-guessing while texting can have really weird results because the guess of what you meant to say has been turned into real words which creates new meaning.
Where previously you’d just have had a few mangled words it’s now been corrected into proper words, often with an unfortunate sexual innuendo.
Machine learning is educated guessing based on previously seen data. As mentioned by others there are ways to do super resolution that only uses the data available. I can't think of any that can upscale a single image, although I have vague memories of having seen something about using moiré patterns to infer the higher resolution texture of some features.
It is especially something that some cameras can do by deliberately doing sensor shifting.
They should not call it super resolution or at best emulated super resolution or artificial super resolution.
Who knows how this evolves and what new applications people may devise? For today, I agree: it's just art.
>To be clear, this isn’t a knock on the Gigapixel software. Vaarakallio tells PetaPixel that the software is “amazing” and he uses it all the time.
Professional photo retouching is art. It's okay to use Gigapixel for an artistic task, it's not OK to use it to enhance a photo that you're going to show to a jury. That's what GP means by 'critical': use cases where it matters whether or not the pixels being added map to an objective reality rather than an algorithmic guess about what would look good.
>you may want to uncheck detect faces… unless you want Ryan Gosling popping up all over the place.
Sooo not really a case against super-resolution, just a funny result of having used the wrong settings