Ask HN: Is there any way that using pinch-to-zoom could change/add pixels? | Hacker News Reader

Ask HN: Is there any way that using pinch-to-zoom could change/add pixels?

16 pointsbeervirus19 comments

Context is here, talking about using an iPad to zoom in on video. [0] My initial thought is no, it just makes existing pixels bigger and blurrier. But then I saw this expert’s testimony[1] and I’m wondering if I’m missing something: “Wisconsin crime lab employee James Armstrong testified, under questioning from defense attorney Corey Chirafisi, that the software program adds pixels to the image and he cannot say with certainty what color the added pixels are.”

[0] https://arstechnica.com/tech-policy/2021/11/rittenhouse-trial-judge-disallows-ipad-pinch-to-zoom-read-the-bizarre-transcript/

[1] https://apnews.com/article/kyle-rittenhouse-technology-wisconsin-kenosha-homicide-b561bef68dc6aadaadc9b45a1bd93a19

View on HN

Yeah, they could be running a sharpening algorithm. Normally you have something between nearest-neighbor resampling and a smooth gradient. But you can plug any function in there if you think it looks good. Bicubic makes nicer smooth areas, sinc can preserve sharp edges. But you can't add information to an image. It's all basically guessing which transitions between pixel values should be smooth and which should be sharp. It's possible to oversharpen an image and generate extra lines out of thin air.

(I don't think pinch-to-zoom adds sharpening, but it's something that could use an expert to double check.)

Here's an image, displayed at "normal" resolution. One pixel of source image goes to one pixel on the display.

Now we zoom. We're displaying a smaller part of the source image, but putting it on the same number of destination pixels. So where do the extra pixels come from? There are (at least) two possible answers:

1. We repeat source pixels across multiple display pixels. This leads to aliasing - to blocky stair-steps in the displayed image.

2. We make up values for the extra display pixels that were not in the source image. This is done by interpolation, not just by random guessing. Bicubic interpolation is pretty good. But still, the program is in fact "making up" values for the new pixels.

Even simpler: a 10x10 image is taken and displayed on a 10x10 screen. The upper right 3x3 pixels are zoomed in on to fill the 10x10 screen.

Now each 3 pixels are being shown on 10 pixels. Each pixel is being spread across 3 1/3 display pixels. How does that work?

For each pixel on the display, you derive a (not necessarily integer) pixel coordinate on the original image. The top-left display pixel may be at (0, 0) on the image. The next pixel to the right is at (0, 0.333) on the image.

So say you're dealing with the pixel at (0, 1.333). You take the pixels at (0, 0), (0, 1), (0, 2), and (0, 3), and you run a cubic interpolation to find the value at 1.333. (You do this three times, for red, green, and blue).

If you're at non-integer coordinates in both directions, then you run an interpolation that is cubic in both directions (that is, a bi-cubic interpolation).

True, but you’ve just introduced new colors!

If you scale the image to add more pixels, you have to decide what the value of those new pixels are in some way. I do not know the methods used by the tools in question. However, the relevant docs for GIMP are here: https://docs.gimp.org/en/gimp-tools-transform.html#gimp-tool...

The real issue here is that this is evidence in a homicide trial where you need accurate evidence to prove beyond reasonable doubt that the events really happened the way they're argued by the parties and it's just not possible in some cases.

To try to answer the question, yes it does/could. Although it depends much on the interpolation algorithm used to enlarge the image or if there was another process that could produce significant alteration such as an AI assisted process.

If you use interpolation to "make it clearer" you're adding information to it and it stops being an accurate representation of the original. It might look clearer to you and it could help you gain more insight, but after the enhancement it's no longer an accurate representation of the original. The very meaning of the word interpolation is "the insertion of something of a different nature into something else."

As an example imagine there's a far away video of someone handing a phone to another person and you take the best frame you can for analysis. In the original it looks like that person blur is handing something to the other person but it's just a blackish line.

After processing with different algorithms in one it now looks like the person is pointing a gun, in the other shows an empty hand and the AI one for some reason shows it's a bag, and so on.

More specifically about the trial the expert witness declared that he does not actually know how the algorithms used work and that he didn't compare with the original. Which I personally find baffling because this is a laboratory that analyzes evidence that may decide the future of a lot of people's life. The technician needs to know how it works in order to be able to reach an objective conclusion that what they're presenting is accurate.

Under the Federal Rules of Evidence (referenced elsewhere), photo manipulation by CSI technicians is allowed, so long as those technicians can be there to attest to the methods and algorithms they used, and they can show the before and after steps at each stage and explain exactly what happened during that stage. This retains the digital chain of custody for the image.

This could easily be used to digitally zoom in on an image, for example.

Where things went sideways here is the prosecution expecting to be able to use pinch-to-zoom as testimony in court, when they should have actually gone through the standard CSI process of producing a digital zoomed image.

Now, to be honest, I think they should have been able to use pinch-to-zoom. If they lose the case because of this problem, I hope they can get a mistrial declared and then go back with the proper procedure.

Otherwise, the prosecution just plain screwed themselves over.

It really depends on what scaling algorithms you use.

Some people can make paintings that look an awful lot like a photograph, that is, they have a mental model of what scenes look like and can construct an image from that model.

Computers can create photorealistic images using raytracing techniques and also with neural networks

https://deepai.org/machine-learning-model/text2img

It's very possible a scaling algorithm could guess at what is missing in the picture and fill something in. That doesn't mean that is going on with Apple products in 2021.

A defense lawyer gave an example - if you have red pixels and blue pixels, would the added pixels between be purple, a color not present in the original image?

The answer is: bicubic, yes; bilinear: yes; nearest neighbor, no.

The visual cortex does image interpolation for crying out loud! Are we supposed to rely on a teleprompter for clarification of what we're looking at? This is a ridiculous argument. Yes, bicubic interpolation primitively aliases between pixels. The human brain does this in much more dramatic ways.

What he is referring to _could_ mean aliasing, where n^2 pixels that can't be accurately represented are merged into m^2 pixels of average colors and values, where m<n.

Still a stretch, and quite a big one at that, so its probably not it.

In what context did this show up in the Rittenhouse trial?

https://arstechnica.com/tech-policy/2021/11/rittenhouse-tria...

That sounds dumb, the judge is either being played expertly by the defense or is blatantly in favor of the accused (the trump rally song ringtone thing).

What a show, I can see why they broadcast it all over the place.

"That sounds dumb"

At first I thought the same thing, but I'm not sure about that. It seems the main point is that the prosecution is required to prove that the manipulation by digital zoom doesn't alter the image in any other way. That seems reasonable. If Apple has a patent, then the methods should be explained enough to show exactly what is being manipulated. If it's not patented/documented, then I think it's reasonable to question what is going on within the black box.

https://www.law.cornell.edu/rules/fre/rule_1001

Federal Rule of Evidence 1001 (e): A “duplicate” means a counterpart produced by a mechanical, photographic, chemical, electronic, or other equivalent process or technique that accurately reproduces the original.

https://www.law.cornell.edu/rules/fre/rule_1002

Federal Rule of Evidence 1002: An original writing, recording, or photograph is required in order to prove its content unless these rules or a federal statute provides otherwise.

https://www.law.cornell.edu/rules/fre/rule_1003

Federal Rule of Evidence 1003: A duplicate is admissible to the same extent as the original unless a genuine question is raised about the original’s authenticity or the circumstances make it unfair to admit the duplicate.

The defense argued that a zoomed-in video is not a duplicate under the rules because the zooming algorithm changed the pixels. If that is true, then the zoomed-in video can't be used as evidence and you have to use the original. If it is false, then the zoomed-in video can be used as evidence.

That's a real argument and the prosecution was unprepared to defend it. If you are going to blame someone (hint hint), don't blame the judge.

Agree. One other thing to point out though, this is a state trial, so I assume there could be minor differences compared to these federal rules.

Oh yes, you are correct. But seriously, pinch-to-zoom has been around for 10 years now, right? This honestly cannot be the first time it's "reproduction accuracy" has been questioned, can it??? The judge gave him half an hour to go do some legal searching, and he came up with nothing??? Milwaukee is just up the road and has a university with a pretty decent CS department. Nobody there was willing to say "yeah, the interpolation algorithm will make an accurate reproduction"? If the prosecution said "I have a professor willing to testify, can we have a delay?" I'm pretty sure the judge would have said yes. Or at least said no so there is something to appeal later on.

It is a little odd there's no prior evaluation. Although, in my experience things like this get glossed over.

A question I didn't see talked about is did they think zooming in would help? My experience has generally been that digital zoom doesn't really help in most cases, especially for distant shots (which if looked like they were viewing).

Reply on news.ycombinator.com