Super Resolution

There are serious use cases for super resolution in medical imaging for example.

Unfortunately someone is going to wrap up super-resolution for critical tasks and sell it likely causing many people harm or at least inconvenience. I have already tried to talk some companies out of using it for police/surveillance type work. People who do not understand the technology are determined to use it and someone is going to.

What I find interesting about that is that after seeing the face in the super-resolution image you can kinda see it in the original

Every time someone brings up enhancing images I like to pull up this Red Dwarf clip:

https://www.youtube.com/watch?v=2aINa6tg3fo

I wish Adobe would use a different name for this that made it more obvious what was happening, something like "detail fill" or "detail interpolation".

I worry this is going to be a case where the marketing is at direct odds with public education efforts.

There are forms of super-resolution that certainly aren't guessing. For example, you can take a video of a subject and integrate over time, so that the motion of the subject over the sensor allows you to infer sub-pixel detail.

https://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResoluti...

Is this single example of someone using different software (by Topaz Labs) relevant to this article specifically? Or just every article about enhancement?

>Photographer Jomppe Vaarakallio has been a professional retoucher for 30 years…

>To be clear, this isn’t a knock on the Gigapixel software. Vaarakallio tells PetaPixel that the software is “amazing” and he uses it all the time.

One of those annoying things is that the name "super resolution" stuck, here.

Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems. Some modern photographic systems actually do this directly (piezo motors?) on the sensor.

Of course this only works if what you are imaging is reasonably static over the time needed to take all the images.

You can do an approximate version of this with video, with caveats because you don't control the motion. The key thing is, though, you actually have more data to work with.

This idea ran in parallel with image processing people attempting to estimate higher resolution from a single image for a while, and unfortunately the terminology stuck in image processing also. Something like resolution extrapolation is probably better but that ship sailed ages ago.

Is there any proof that Ryan Gosling's face (or perhaps a photograph of Ryan Gosling's face) was in fact not there when the original photo was taken? :)

Lots of things are "only guessing." Auto color correction is only guessing. Unsharp mask is only guessing. Smart selection is only guessing. Content-aware fill is only guessing.

They're still useful tools to have in your toolbox as a photographer or designer, even for critical tasks, and I don't really see how this is different. There may be certain failure cases, but everything has failure cases.

For me super resolution mean a combination of multiple lower resolution images to gain additional information and from that higher resolution.

It is especially something that some cameras can do by deliberately doing sensor shifting.

They should not call it super resolution or at best emulated super resolution or artificial super resolution.

Guessing I think is too strong of a way to phrase what super resolution does. The broader concept of, for example, regularized solving of inverse problems is used widely in things like CT and MRI where the reconstructed imagery is used for analysis. The regularization is effectively the part you’re saying is guessing, but I would phrase it as enforcing assumptions about the data. Neural network-based approaches are similarly learning the distribution of the output data.

From the linked article:

>you may want to uncheck detect faces… unless you want Ryan Gosling popping up all over the place.

Sooo not really a case against super-resolution, just a funny result of having used the wrong settings

> Super-resolution is only guessing.

Machine learning is educated guessing based on previously seen data. As mentioned by others there are ways to do super resolution that only uses the data available. I can't think of any that can upscale a single image, although I have vague memories of having seen something about using moiré patterns to infer the higher resolution texture of some features.

I feel like it's also ok for critical tasks if you're willing to accept that it isn't perfect. If all you have is a grainy photo, you'll only be able to make guesses yourself; why not have a superhuman guess too? (Because the people putting it to use would be morons about it, I know, let me dream)

Or https://twitter.com/Chicken3gg/status/1274314622447820801

Nitpicky, but I think "It's okay for x, not for y" when describing nascent technology is a bit shortsighted.

Who knows how this evolves and what new applications people may devise? For today, I agree: it's just art.

A personal favourite is white Obama: https://www.theverge.com/21298762/face-depixelizer-ai-machin...

Nobody has billed it as anything more than just guessing. In the literature, it is frequently mentioned as “perceptually plausible” upscaling.

Example of critical tasks?

That reminds me of the old CSI episodes where they'd have grainy footage from a CCTV running around SIF/240p, and the lead investigator would say "Enhance... Enhance... Enhance... There!" And the face of the killer would be clear as day from inside a moving car a block away.

Wow! The 3D-Sim of the nuclear envelope at the wikipedia article was amazing but seeing the structure in the neuron is astounding. Do you know how long imaging takes for this? I assume the post-processing is slow but will video be possible someday?

Yes, I'm disappointed by the name collision. Is there a tool that makes "real" superresolution photos easy? Is that built into photoshop as well under a different name?

Another "real" technique I've seen is having sensors that do not have the individual phototransistors layed out in a nice periodic grid pattern but with aperiodic pattern like Penrose tiling (put PTs as vertices of kites/darts).

The interpolation techniques (making a photo out of hardware input) manage to get 10-15x more resolution out of that sensor layout compared to normal grid.

You also avoid Moire patterns with that too.

They have a tool specifically crafted to remove their buggy software:

https://helpx.adobe.com/creative-cloud/kb/cc-cleaner-tool-in...

Note that this isn't the same as uninstalling everything via the official process since this leaves behind stuff like Adobe's Genuine client which verifies you're not using pirated software.

It's not only Creative Cloud itself but all the newer versions of their apps I recently used.

I needed to update my CV recently and expected to spend 1h in InDesign. I spent 6h in the end.

– InDesign crashes while saving and destroys my document. 1h lost.

– InDesign crashes while exporting a PDF of my document (9 pages). I hadn't saved. 1h lost.

– InDesign crashes (reproducible) when adding/inserting a page (mind you, that's page 10). First time this happened I hadn't saved for half an hour. I was really considering changing the text because I couldn't solve this. Then I found [1]. Quote:

> [...] after speaking to Adobe chat help, they asked me to send my file to them. They sent it back to me and everything went back to normal. [...] "File was corrupted , we recovered it by using scripts and then saved as IDML."

– Because of the above I had the idea of exporting to IDML. Re-importing then allowed me to add the page but I had subtle formatting errors where the last character before a tab or a newline on lines that had the font changed via a character style had the wrong style. Fixing this: 1h.

– When I re-arranged parts of the CV via copy & paste entire sections I copied lost the small caps/italic styles they had assigned (acronyms/names). Going through the entire document to fix this: 1.5h.

I should have known better. Less than two years ago I helped a friend do a snail mail mass mailing where we used a CSV file with addresses to create hundreds of (two page) letters. All in InDesign. Everything worked until we tried to export as PDF, for printing. The solution was to export as 'interactive' PDF and only export about ~100 pages at a time.

I bought Affinity Publisher already when the thing with the letters happened. But I naively believed updating my CV would be quick in InDesign.

In retrospect typesetting the CV from scratch in Publisher would have been the better choice.

Last week I helped a friend with a commercial that was mostly 3D and some motion graphics done in After Effects (Ae). We couldn't get it to render in After Effects 2019. It would run out memory and then just not render the frame or crash. In the end we exported the project for an older version and went back to an Ae CC version from six years before. That worked without any issues.

All this is just shocking. I used InDesign from 1.0 and it was not that bad, a decade ago. Ae ... the same. See above.

As of a recent update, Acrobat Reader (free version) refuses to let me open any document w/o signing into CC first. Another wtf.

What a friend of mine replied when he heard about my InDesign adventure:

> I'm on CS6 for anything Adobe. Just junk now.

[1] https://community.adobe.com/t5/indesign/indesign-crashes-whe...

People are doing this to themselves. Stop buying their shitty software, and see how quickly they start to fix it.

I don't understand why people are so obsessed with Adobe, since their software nowadays isn't that good. There are tons of alternatives out there that work better and do the same thing, if not more.

Is it just laziness/reluctance to learn something new?

I agree with this. I’ve used the Pixelmator Pro feature a lot and it’s damn good.

I hear it’s just monstrously fast on the M1 too.

It looks like the character changed into a silk version of his shirt with no chest pocket.

That’s cool, thanks for sharing. How long does a conversion like that usually take?

Videos can use inter-frame information to help infer sub-pixel details.

This post about "Super Resolution" is interesting because it starts with RAW format (which contains information about camera sensor arrangements), hence, the machine-learned model should not only memorize artificial details (what hair should be look like, what a tree-leaf should be look like etc, and use that to "hallucinate" a higher-resolution details, I liked to call this "hallucination" for that reason), but also relationship between complex interference of different sensors in their corresponding arrangements.

You can read more about RAW format and why exposing RAW format for photography is exciting (on everyday's camera, i.e. your phone) from this post: https://blog.halide.cam/understanding-proraw-4eed556d4c54

I might be old and grumpy, but I prefer the left image to the right one? The right one is watercolory and overly smoothed, like a cheap beautify photofilter

>Any combination of lenses and film stocks (or equivalent) is going to represent but a flat, skewed representation of the three-dimensional world

this is no different than any other sensor. It doesn't mean replacing data with guesses is better than the sensor representation of the world, which is the issue at hand today.

more or less the same thing with sound and audiophiles. it's a lucrative business of selling placebos, dreams and hope, though.

Anyone who edits photos in Lightroom understands that modern photography captures both reality and artistry.

My goal when editing photos is to make them more clearly express how I felt or how I saw. This is often quite divorced from what shows up on the back screen of my camera.

A good experiment (which I cannot do): take a true photo of somebody, reduce it in size and then “super-resolve” it. How much must you reduce in order to produce a different “person” (or a non-person).

Yeah, although technically the goal behind things like 'super resolution' is to add data that is not there anymore but probably was there before.

i.e. you had the original scene that was captured by a digital camera (a lossy operation) and then saved as an image file (often also a lossy operation), and then a tool like this makes an educated guess as to what information was lost in the 1st and 2nd steps.

Photography has always been more artistry than reality. From film selection, lens choice, framing and cropping, color correction, "dodging" and "burning" details in our out it's always been a compromise between what the camera sees (which itself may not be reality if you're using physical in-front-of-the-lens filters or other distortive techniques) and what the photographer want to convey. That's one of the reasons that Photographic Journalism often holds itself to ethical limitations with regards to how a photo can be manipulated after it's been taken. I'm almost certain that this kind of thing would at least walk that fine line if not fall right on over it.

It's always been more like painting than anything resembling a truthful representation of reality. In fact, the first ever photograph [1] much more closely resembles a painting than anything else, which is ironic.

All advancements have simply given us more control in how painterly we render our photographs, but have never _really_ brought us closer to the truth.

[1]: https://en.wikipedia.org/wiki/History_of_photography#/media/...

Ansel Adams answering a similar question https://youtu.be/Ml__B0l9GIs?t=1514

My worry is when this gets used for something forensic.

I imagine this will tend to reproduce things in the dataset, e.g. up-scaling blurry text may look like fonts that it has memorized more than the original. Or upscaling a feather will provide details like the feather of more common birds. Or upscaling blured out numbers will pick some numbers at random [1].

We need to make sure people don't rely on these details, e.g. in courts, HR reviews, when reddit sleuths try and investigate an incident, when someone looks for cheating partners etc.

[1] https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...

It is sort of. Their argument is that by taking into account the subpixel pattern of the sensor they can actually extract more detail than is readily visible in the picture.

Basically you can imagine that the blue subpixel is always to the top-left of the pixel. If you shifted the blue down and right one half a pixel you would have a more "accurate" production. In this way you can add a new pixel with the blue value closer to the right spot, then interpolate the blue of the original.

Of course you can also do logic such as detecting lines of different lightness and applying those on top.

So yes, especially with their machine learning they are adding new detail, but that is also likely some detail that was already there, but could not be conveyed with with the lower resolution. I wonder how different this would be from the simple approach of realigning the subpixels on a higher-resolution image and interpolating the "missing" subpixels. This approach may look better but wouldn't add any data.

Not true, there’s always far more data in an image than a quick glance can decipher, astronomy field obviously pioneered numerous methodologies over the decades (many of which biologists do more shittyly, speaking as a biologist who did so). In the end I’d argue a lot of these “super resolution” methods are just glorified dexonvolution, not that it’s a knock on these methods but apparently it’s not cool to call deconvolution deconvolution anymore.

Presumably you go from picture to painting the moment you open photoshop.

Photography for record keeping/science and photography for aesthetics/artistry diverge long before you get to techniques like super resolution. Which is still a fuzzy boundary because anyone who has taken a picture including a sunny sky can tell you raw photos generally don't capture how it looks to your eyeballs.

Back in the day when they first had their sub model it was soooo much cheaper than it is now :(. Here's hoping some guy that writes super-resolution ML models on the side is a big fan of Gimp.

Who has old 800x600 camera raw images lying around?

Looking at results so far, it's strictly better. (Less noisy, less subjective errors on fine detail)

If I want to up-rez strictly for printing purposes, the PS one looks like the winner. But, obviously, it's subjective.

I enjoyed this review https://stephenbayphotography.com/blog/adobe-super-resolutio...

Both of the options had winners.

Does it have to be? I think the point is it's built into Adobe products now, which will be fantastic for Photoshop/Lightroom etc users.

Unless ofc you're genuinely curious as to how it's different to Gigapixel and not just knocking Adobe :)

True. Our brain also makes similar guesses, e.g. in the blind spot, and in most optical illusions.

Vaguely relatable, there is/was stuff like Photosynth [1] which (as far as I recall, from researching this stuff as a CS undergrad) was making use of Bundle Adjustment [2] and detecting landmarks in images using Scale Invariant Feature Transform (SIFT) [3], to estimate camera poses of photos relative to each other in a 3D scene -- even from completely different cameras.

By projecting the discarded frames onto your keeper frame, you've potentially got multiple samples of the same pixels and their neigbours...

(Although I guess the "every image is a plane" 3D transform is a bit simple and doesn't account for lens distortion)

[1] https://en.wikipedia.org/wiki/Photosynth [2] https://en.wikipedia.org/wiki/Bundle_adjustment [3] https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...

Well, is it a hobby, or is it family documentary? Feels like similar sentiment if you said that money spent on woodworking gear is wasted because family prefers IKEA furniture.

Right. I wish somebody would do a proper technical analysis of this instead of all these current "well the super-res version looks good to me" reports.