A reality bending mistake in Apple's computational photography
appleinsider.com
appleinsider.com
[0]: https://petapixel.com/2023/11/16/one-in-a-million-iphone-pho...
Is it 100% pixel perfect to what was happening at the time? No, but I also don’t care.
I’ve used HDR exposure stacking in the past. I’ve used focus stacking in the past for shallow depth of field. I’ve even played with taking multiple photos of a crowded space and stitching them together to make an image of the space without any people or cars. None of them are pixel perfect representations of what I saw, but I don’t care. I was after an image that captured the subject and combining multiple exposures gets the job done.
No photographer thinks images the they get on film are perfect reflections of reality. The lens itself introduces flaws/changes as does film and developing. You don't have to be a purist to want the ability to decide what gets captured or to have control over how it looks though. Those kinds of choices are part of what make photography an art.
In the end, this tech just takes control from you. If you're fine with Apple deciding what the subject of pictures should be and how they should look that's fine, but I expect a lot of people wont be.
Don't fall into this trap. A lens and computational photography are not alike. One is a static filter, doing simple(ish) transformation of incoming light. The other is arbitrary computation operating in semantic space, halfway between photography and generative AI. Those are qualitatively different.
Or, put another way: you can undo effects of a lens, or the way photo was developed classically, because each pixel is still correlated with reality, just modulo a simple, reversible transformation. It's something we intuitively understand, which is why we often don't notice. In contrast, computational photography decorrelates pixels from reality. It's not a mathematical transformation you can reverse - it's high-level interpretation, and most of the source data is discarded.
Is this a big deal? I'd say it is. Not just because it rubs some the wrong way (it definitely makes something no longer be a "photo" to me). But consider that all camera manufacturers, phone or otherwise, are jumping on this bandwagon, so in a few years it's going to be hard to find a camera without built-in image-correcting "AI" - and then consider just how much science and computer vision applications are done with COTS parts. A lot of papers will have to be retracted before academia realizes they can no longer trust regular cameras in anything. Someone will get hurt when a robot - or a car - hits them because "it didn't see them standing there", thanks to camera hardware conveniently bullshitting them out of the picture.
(Pro tip for modern conflict: best not use newest iPhones for zeroing in artillery strikes.)
Ultimately you're right, though: this is an issue of control. Computational photography isn't bad per se. It being enabled by default, without an off-switch, and operating destructively by default (instead of storing originals plus composite), is a problem. It wasn't that big of a deal with previous stuff like automatic color corrections, because it was correlated with reality and undoable in a pinch, if needed. Computational photography isn't undoable. If you don't have the inputs, you can't recover them.
Artistic interpretation of a scene is often very nice.
But we would really need to be able to discern cameras that give you the pixels from the ccd from the irreversible kind.
In the worst case scenario it’s back to photographic film if you want to be sure no-one is molesting the data :D
I mean… you’ve pretty much described our brain. Your blue isn’t my blue.
People need to stop clutching pearls. For the scenario it is used in, computational photography is nothing short of magic.
> In the worst case scenario it’s back to photographic film if you want to be sure no-one is molesting the data :D
Apple already gives you an optional step back with ProLog. Perhaps in the feature they’ll just send the “raw” sensor data, for those that really want it.
>Or, put another way: you can undo effects of a lens, or the way photo was developed classically, because each pixel is still correlated with reality,
You cannot undo each and every effect. Polarizing filters (filters, as are lens coatings, are part of a classical lens in my opinion), gradual filters, etc effectively disturb this correlation.
As does classic development, if you work creatively in the lab (as I did as a hobby a long time ago in analog times) where you decide which photographic paper to use, how to dodge or burn, etc.
But yes, I agree that computational photography offers a different kind of reality distortion.
Fair enough.
> Polarizing filters
Yeah, I see it. This one is as pure signal removal as it comes in analog world. And they can, indeed, drop significant information - not just reflections, but also e.g. by blacking out computer screens - but they don't introduce fake information either, and lost information could in principle be recovered -- because in reality, everything is correlated with everything else.
> But yes, I agree that computational photography offers a different kind of reality distortion.
A polarizing filter or choice of photographic paper won't make e.g. shadows come out the wrong way. Conversely, if you get handed a photo with wrong shadows, you not only can be sure it was 'shopped, but could use those shadows and other details to infer what was removed from the original photo. If you tried the same trick with computational photograph, your math would not converge. The information in the image is no longer self-consistent.
That's as close as I can come up to describing the difference between the two kinds of reality distortion; there's probably some mathematical framework to classify it better.
Auto-censoring camera, if you like.
(Also don't try to point such camera at your own kids, if you value your freedom and them having their parent home.)
Oh yes you can, and the black helicopters will be dispatched, your social score obliterated and credit ratings becomes skulls and bones. EU Chat Control will morph to EU Camera Control. Think of the children!
No, you most definitely cannot. The roots of computational photography are in things like deblurring, which in general do not have a nice solution in any practical case (like non-zero noise). Same deal with removing film grain in low light conditions.
It’s absolutely not reversible. Information gets lost all the time with physical systems as well.
Also, probably something like the creation of the image of the black hole is closer to computational photography than “AI”, and it seems a bit like yours is a populist argument against it.
I do have reservations about that image, and don't consider it a photograph, because it took lots of crazy math to assemble it from a weak signal, and as anyone in software who ever wrote simulations should know, it's very hard to notice subtle mistakes when they give you results you expected.
However, this was a high-profile case with a lot of much smarter and more experienced people than me looking into it, so I expect they'd raise some flags if the math wasn't solid.
(What I consider precedent to highly opaque computational photography is MRI - the art and craft of producing highly-detailed brain images from magnetic field measurements and a fuck ton of obscure maths. This works, but no one calls MRI scans "photos".)
In these difficult scenarios, the alternative photo I'd get using such a small camera without this kind processing would be entirely unusable. I couldn't rescue those photos with hours of manual edits. That may be "in control", but it isn't useful.
I mean, at a certain point taking a less than perfect photo is more important than getting a fake image that looks good. If I see a pretty flower and want to take a picture of it, the result might look a lot better if my phone just searched for online images of similar flowers, selected one, and saved that image to my icloud, but I wouldn't want that.
But "in difficult scenarios", as the GP comment put it, your mistake is assuming people have been taking those photos all along no problem. They have not. People have been filling their photo albums and memory cards up with underexposed blurry photos that look more like abstract art than reality. That's where this sort of technology shines.
I'm pretty reasonable at getting what I want out of a camera. But at some point you just hit limitations of the hardware. In "difficult scenarios" like a fairly dark situation, I can open the lens on my Nikon DLSR up to f/1.4 (the depth of field is so shallow I can focus your eyes while your nose stays blurry, so it's basically impossible to focus), crank the ISO up to 6400 (basically more grain than photo at that point), and still not get the shutter speed up to something that I can shoot handheld. I'd need a tripod and a very still subject to get a reasonably sharp photo. The hardware cannot do what I want in this situation. I can throw a speedlight on top, but besides making the camera closer to a foot tall than not and upping the weight to like 4lbs, a flash isn't always appropriate or acceptable in every situation. And it's not exactly something I carry with me everywhere.
These photos _cannot_ be saved because there just isn't the data there to save. You can't pull data back out of a stream of zeros. You can't un-motion-blur a photo using basic corrections.
Or I can pull out my iPhone and press a button and it does an extremely passable job of it.
The right tool for the right job. These tools are very much the "right" tool in a lot of difficult scenarios.
Not in these light conditions. Simple as that. What iPhones are doing nowadays gives you the ability to take some photos you couldn’t have in the past. Try shooting a few photos with an iPhone and the app Halide. It can give you a single RAW of a single exposure. Try it in some mildly inconvenient light conditions, like in a forest. Where any big boy camera wouldn’t bat an eye, what the tiny phone sensor sees is a noisy pixel soup that, if it came from my big boy camera, I’d consider unsalvageable.
That is a bit harsh. The vast majority of all people are worse at photography than apple's algorithm.
GP> In the end, this tech just takes control from you.
In the end, any tech is always going to take control away from you one way or another. That’s the whole point of using it, so you can achieve things you wouldn’t otherwise be able to.
It might not be a conformal mapping but, say, if the image of a grid is projected through the lens, the projected image will have the same number of divisions and crossings, in the same relation to each other.
We cannot say this about an AI-powered transformation which changes a body posture.
> What counts as “photography” really?
Focusing light via a lens into an exposure plate, from which an image is sampled.
Life is imperfect. Lens specs are only good for reviews / marketing because it is hard to otherwise compare lenses for there real value.
I am fine with control I am getting from the big camera.
[1] https://gebseng.com/media_archeology/reading_materials/Bob_S...
Ah yes, for all those people using their iphone to make true art instead of for taking a selfie.
The market for cell phone cameras are generally casual users. High end artists generally use fancy cameras. Different trade offs for different use cases.
that is very due to camera makers not giving a sh*t though. Maybe now they do, but it's overdue. And I'm not even talking about the phone apps for connection, or image processing, but more on a user interface and usability PoV, and also interesting modes for the user.
Fuji and Ricoh are the only ones that I see are trying to make things easier or more fun or interesting for non-professionals. Fuji has the whole user customization that people use for recipes and film simulation (on top of the film simulations they already have), and Ricoh is the only one (I know of) that has snap focus, distance priority, and custom user modes that are easy to switch to, and to use. But even Fuji and Ricoh could still improve a lot, since there's always a detail or another that I'm like... why did they made it like that? or... why didn't they add this thing?
It’s obviously an argument to say that Apple shouldn’t get to choose how your memories are recorded, but I know I’ve captured a lot more moments I would’ve otherwise missed because of just how automatic phone cameras are.
There’s a place for both kinds of cameras IMO.
I'd argue that the women trying to take a photo of herself in her wedding dress did not get a clear picture that captures a memory. She got a very confusing picture that captured something which never happened. There are lots of great automatic camera features which are super helpful, but don't falsify events. If I take a picture of my kid I want my actual child in the photo, not a cobbled together AI generated monstrosity of what apple thinks my kid ought to have looked like in that moment.
Automatic cameras are great. Cameras that outright lie to you are not.
Oh the irony of framing things (pun intended) so hyperbolically. Somehow it never seems to dawn on people that like to throw around the word ‘lie’ that they’re doing exactly what they’re complaining about, except intentionally, which seems way worse. Nobody sat down to say bwahahah let’s make the iphone create fake photos, the intent obviously is to use automated methods to capture the highest quality image while trying to be relatively faithful to the scene, which might mean capturing moving subjects at very slightly different times, in order to avoid photo-wrecking smudges. When you blatantly ignore the stated intent and project your own negative assumptions of other people’s motivations, that becomes consciously falsifying the situation.
Photographs are not and never have been anything but an unrepresentative slice of a likeness of a moment in time, framed by the photographer to leave almost everything out, distorted by a lens, recolored during the process, and displayed in a completely different medium that adds more distortion and recoloring. There is no truth to a photograph in the first place, automatic or not, it’s an image, not reality. Photos have often implied the wrong thing, ever since the medium was invented. The greatest photos are especially prone to being unrealistic depictions. Having an auto stitch of a few people a few milliseconds apart is no different in its truthiness from a rolling shutter or a pano that takes time to sweep, no different from an auto shutter that waits for less camera shake, no different from a time-lapse, no different from any automatic feature, and no different from manual features too. Adjusting my f-stop and focus is really just as much distorting reality as auto-stitching is.
Anyway, she did get a clear memory that was quite faithful to within a second, it just has a slightly funny surprise.
Isn't driving while blind illegal?
> the best camera is the one you have with you
> I’ve captured a lot more moments I would’ve otherwise missed
Soon these moments will be 'captured' so perfectly with simulation that there wouldn't be reason taking them. Generate them when you want to recall them, not when the moment is happening.
JIT 'photography'
At this point, photography becomes a shittier version of what our brains do, so why bother? Or maybe let's do that, but then let's also do the kind of photography that accurately represents photons hitting the imaging plate.
Don’t care how it happens I don’t want to think about settings.
There's no "objective photography", no matter how hard we try distinguishing between old and new tech.
More importantly than that: framing, exposure, aperture choices by anyone not simply shooting with their camera in the Auto mode.
* https://petapixel.com/digital-camera-modes-a-complete-guide/...
* https://en.wikipedia.org/wiki/Mode_dial
* https://photographylife.com/understanding-digital-camera-mod...
This is a non-issue.
ISO 100, f/11, manual focus at infinity, shutter speed 1/100.
Phones aren't good at this, partly because autofocus isn't designed for moons, partly because you want a really long lens and they don't have one.
The majority dont care, as the majority are not photographers, nor is it meant to be a product for photographers. Average joe just wants a good photo of that moment, which it does exceptionally well.
The tech 'taking control from you' is its exact purpose, as again, not everyone is a photographer. The whole point is to allow 'normal people' to get a good photo at the press of a button, it'd be incredibly foolish and unreasonable to expect them to faff about with a bunch of camera settings so it does it for you.
1. "This tech" is too broad of a qualifier, as GP is talking about computational photography in general, which is many things at once. Most of those things work great; some others are unpredictable. There are plenty of custom camera apps besides Apple's default camera which will always try to stay as predictable as possible.
2. There is such a thing as too much control. Proper automation is good, especially if you aren't doing a carefully set up session. The computational autofocus in high-end cameras is amazing nowadays. You can nail it every time now without thinking, with rare exceptions.
It’s just picture of my dog. It’s not that serious.
Until you want to send it to your vet, but you can't, because your phone keeps beautifying out the exact medical problem you're trying to image.
Having control isn't always the best option. While photographers may appreciate having a camera over which they can have a high degree of control, they're not snobs about it. Photographers will tell you that having a handy point and shoot with good automation or correction of some kind is extremely useful when you're out and about and need to take a photo quickly. For everyday photos, it's what you'll want usually. If not, there are plenty of cameras on the market to choose from.
Can't this be disabled?
And can't some other product with a similar technology offer knobs for configuring exactly how it behaves?
I’m reminded of that scene in WALL-E where it shows the captain portraits as people get fatter and fatter. It’s clearly inaccurate: over time the photos should show ever more attractive, chisled captains. They’d still be obese in real life though.
I really wonder what's that doing to insecure teenagers with body issues.
When you talk to a client, a colleague or a loved one we’re on the verge of you conversing with a model that mostly represents their image and voice (you hope). The affordances of that abstraction layer will only continue to deepen from here too.
Media processing is experiencing a fundamental change though. Traditional compression removes information. What's happening now creates new information and interacts with semantic context.
Wow, this threw me for a loop. There’s not much difference between talking to a heavily filtered image of a person and texting with them. Both the image and the words are merely representations of the person. Even eye-to-skin perceiving someone is a representation of their “person” to a degree. The important part is that “how” and “how much” the representation differs from reality is known to the observer
What we have today already includes phone cameras that will add teeth to the image of your smiling newborn, or replace your image of what looks like the moon with stock photography of the moon.
> I’m reminded of that scene in WALL-E where it shows the captain portraits as people get fatter and fatter. It’s clearly inaccurate: over time the photos should show ever more attractive, chiseled captains.
Interestingly, the opposite thing happened with official depictions of Roman emperors.
>".... if they can detect and preserve fabric, rope, cloud, and other textures, why not skin texture as well?"
The phone changed the image on skin and not on random other things that wasn't a face. That is a filter, not normal image processing, when it happens only on X but not on Y. The only way to not get this filter on the phone is using RAW.
They call it 'instagram look' for quite some time, Apple is the worst among all phone manufacturers (in form of furthest from actual ugly reality, but a lot of people got used to it and actually prefer it now), but all of them are making it.
This is not at all like DALL-e and midjourney, this is literally putting together multiple images taken shortly after another, but instead of dumbly putting it on another as it would manually happen in photoshop with transparent layers, it takes the first photo as a key, and merges info from the other photos into the first where it makes sense (e.g. the background didn’t move, so even this blurry frame is useful for improving that).
This is just dishonest to mix AI into that.
https://www.insider.com/samsung-phones-default-beauty-mode-c...
My favorite one is the phones that put a fake picture of a moon in pictures where the moon is detected!
https://www.theverge.com/2023/3/13/23637401/samsung-fake-moo...
This is just about as wrong as saying Stable Diffusion contains a stock photo of the moon it spits out when you prompt it with "the moon".
They work the same way. Samsung's camera has a moon mode, so it gets the prior (a much, much lower quality camera raw than you think it's getting), it processes it with a bias (this noise is a Gaussian distribution centered on the moon), and you get a result (an image that looks like the moon).
You might remember an article about how there are many situations where the iphone just takes bad portraits, because its idea of what good lightning is breaks down. 5 year old phones often take pictures I like more than one of the latest phones, and not because the hardware was better, but because the tuning is just bad.
Fun things also happen when you take pictures of things that are not often in the model: Crawlspaces, pipes, or, say, dentistry closeups. I've had results that were downright useless outside of raw mode, because computational photography step really had no idea of what it was doing. It's not that the sensors are limited, but that the things that the iphone does sometimes make the picture far worse than in the past, when it took fewer liberties.
A non-purist will hate it too, as soon as the technology re-imagines the positions of his hands such that they are around someone's throat.
Photography as an art was never about purity, and I think most of us want photos that reflect what we see and how we see it, and will take the technical steps to get that rendered. If the moon is beautiful and gently lights the landscape, I want a photo with both the bright moon and shadowy background, and will probably need a lot of computation for that.
But the doppelganger brides, or the over-hdred photos, or the landscapes with birds and pilars removed aren't what someone is seeing. They can be nice pictures, but IMO we're entering a different art than photography.
Now if society (including the courts) learn that the photos might not reflect reality, photo "evidence" could face issues being accepted by courts...
I assume road assistance apps could also have that kind of feature. It's definitely becoming a thing.
You simply have to be practical, use the best took for the job you need. If you ever actually listened to photo artists talking among their peers about their art (ie Saudek), they practically never talk about technical details of cameras or lenses, its just a tool. If they go for analog photography its because they want to achieve something thats easier for them like that maybe due to previous decades of experience, not some elitist Luddites mindset. Lightning of the scene, composition, following the rules and then breaking them cleverly, capturing/creating the mood etc are what interests them.
But it is easy to understand the artists. It is said that in art everyone needs to master the technique first, the tool of the trade, but true works of art are the expressions that are created with these various techniques. At this point, a tool - computational photography in this case - may get in a way. So, it is not about purism. Quite the contrary, it is about being able to use the tools and bend the reality the way an artist wants.
Having said that, I would think anyone would normally use *all* the tools available at their disposal, and the truth is that iPhone camera among else is a great one anyway.
B&W film photography + darkroom printing is an art form, as is digital photography + photoshop. These modern AI assisted digital photography methods are another art form, one with less control left to the photographer, but there's nothing inherently wrong with that. I wouldn't want to say which is better, it's not really an axis that you can use to compare art is it?
At the end of the day, do you generate an image which communicates something that the photographer had in mind at the time? If so, success!
People use their cameras - especially phone cameras - for both these purposes, and often which one is needed is determined after the fact. So e.g. I might appreciate the camera touching up my selfie for the Instagram post today, but if tomorrow I discover a rash on my face and I want to figure out how long it was developing, discovering that all my recent selfies have been automatically beautified would really annoy me.
Or, you know, just try being a normal person and make a photo of your kid's rash to mail/MMS to a doctor, because it's a middle of a fucking pandemic, your pediatrician is only available over the phone, and now the camera plain refuses to make a clear picture of the skin condition, because it knows better.
I'm also reminiscing about that Xerox fiasco, with copy machines that altered numbers on copied documents due some over-eager post-processing. I guess we'll need to repeat that with photos of utility meters, device serial numbers, and even scans of documents (which everyone does with their phone camera today) having their numbers computationally altered to "look better".
EDIT:
Between this and e.g. Apple's over-eager, globally-enabled autocorrect at some point auto-incorrecting drug names in medical documents written by doctors, people are going to get killed by this bullshit, before we get clear indication and control over those "magic" features.
There's the right tool for every job, an actual camera with a nice lens is the tool for the job when you want to "take a photo", situation permitting.
I do. Photos can be material to court cases where people's money, time, even freedom are at stake. They can sway public opinion, with far-reaching consequences. They can change our memories. At the very least, there should be a common understanding of exactly how camera software can silently manipulate photos that they present as accurate representations of reality.
The problem is more one of what controls the camera exposes to the user. If you can just take one kind of picture: whatever picture the engineers decided was 'good', then it limits your expressive options.
I am more in the purist camp, because when people take iOS photo, I remind them that someone else made that decision on how the photo should look. Additionally, we are in an era of not trusting anything on the internet or in a photo anymore. Do we want photojournalism to go that same path? I don't. So I enjoy being closer to "reality" than the computational photos, but for average entertainment photos, I don't mind.
I think I had to spend ~$1k to get my first DSLR with RAW support back in the 2000s. Adjusted for inflation, Halide + a recent iPhone feels like a pretty good deal.
https://photos.app.goo.gl/qChwaw9C29WVdAmr6
If you zoom in, you'll note that .. everything at a detail level looks like an oil painting, especially the deer's face and the wall behind it. Very weird effect, and that's certainly not what the wall or deer actually look like. No filters applied.
Computational photography is really awesome and modestly worrisome.
Basic denoising is stuff like chroma or hue smoothing. This is very aggressive patterning. That's a daylight photo that the phone turned into an oil painting.
https://www.google.com/search?q=pixel+photo+oil+painting+loo...
The noise pattern of any sensor that small will blotch in patterns that when combined with aggressive noise reduction, it will look this way.
It’s less apparent on bigger sensors but you’ll see the same if you really crank the noise reduction
The laws of physics put a limit on how well a tiny phone lens and sensor can capture light. We’re not at the limit yet even though hardware is quite good. Still, you wouldn’t like if your phone spat out raw, noisy images everywhere without any processing (beyond what’s necessary to translate it to the right pixel grid and color space).
Noise does not automatically look worse than this weirdness. It can look bad when resizing with fast algos (moire and all). And of course noise is super fine detail so preserving it blows up filesize.
Even cameras that don't make any intentional decisions end up making decisions because of the physics of the sensor as an electrical device and how you read each sensor element.
You can modify the camera drivers etc on some devices to get nearly no processing output (other than the debayering color filtering etc by the ISP). I custom modded an LG V20 that I use for my mobile photography. I've either totally disabled or adjusted nearly every function of the signal processor as well as modified the camera program itself. I also use a custom modified gcam on it.
Some samples: https://imgur.com/gallery/W42TVGf
Look at my user (helforama) on imgur and pretty much every uploaded photo I took and edited on my V20... I need to update that account with newer photos :) been using the phone aince 2016 for photos!
I ask because the Pixel 7 doesn't have a dedicated zoom camera, so any zoom was digital which reduces quality a lot. The processing part has to use low quality frames and what you have there is often the result.
Phone users shouldn't have to think about this, but from experience I think that on phones without a zoom camera it's often better to take a photo without zoom (or avoid going past 2x) and crop the image afterwards.
A few weeks ago, I took a really lovely picture of my son, composition, facial expression, focus, light, it was _PERFECT_ except..
The algorithms in my phone had decided that it'd be better to scrape off his fucking skin and replace it with the texture of the wall behind him!
Of course it must be my fault for buying such a cheap phone, it only a Galaxy 22 Ultra, I'm sure the 23 Ultra is better... But it was not out when I changed phones..
Wtf^wtf..
So I go turn on RAW so I can at least salvage picture in the future, except RAW only works in the "pro" camera mode which is inconvenient to use and sometimes it silently falls back to non-pro..
In the end I gave up and installed a third party camera application, I guess I just have to trust Mark, at least he hasn't actively messed up my photos. https://play.google.com/store/apps/details?id=net.sourceforg...
Hell, I don't think it's using computational photography besides basic denoising/lightening magic for low light (and their stupid moon replacement thing...they already beat Apple just having the 10x lens, no need for the moon bs).
I think the default camera app is fine.
The white label may appear sharper because white reflects more light back...so higher contrast and less obvious noise.
Contrast is good for denoising algos afaict, textured surfaces are horrendous cause...you can't tell what's noise in the texture so you just gotta do your best.
Every phone will create shots like this in low light/indoors.
We have an inbuilt set of assumptions about causality that this AI now violates. That's potentially huge in some very specific scenarios...
I wouldn't be surprised if a similar thing happened here. Different frames, processing picks the best exposure for each part of the picture and you get this effect.
e: image link https://ibb.co/nwbw5xY
Every other phone just has an algorithm to estimate how much noise is in an image by sampling a few patches - and that will remove snow.
[0] This one, I think: https://podcasts.apple.com/nz/podcast/the-talk-show-with-joh...
If you use the software button you can slide to the left for burst, but afaik there's no way to trigger burst photos from the volume buttons. Maybe the new programmable button on the iPhone 15 series.
"Tip: You can also press and hold the volume up button to take Burst shots. Go to Settings > Camera, then turn on Use Volume Up for Burst."
Historically Live Photos were of poorer overall image quality, so I only turn them on when I want to simulate a long exposure. Not sure whether that's still true.
Some would say only the classical optical camera would capture faithfully our reality. But does it? The reality of the sunlight is a broad spectrum of radio emissions: UV, infrared and more. Does the optical camera capture these? No. Thus, which reality does it capture? Our perceived reality? Other would argue: at least the optical system would capture events in time faithfully. But does it? What would we see in a femto second? Certainly not the pictures we normally see. So the results of an optical system are also super imposed realities, not very much different than the results of a computational photography.
There is simply no single one reality, only our perceived realities. But if so, can we still call it reality or it’s merely a product of our sense, our perception and hallucination?
This does not follow at all from your earlier paragraph.
The reality we're talking about here, which regular photography reflects while computational one doesn't, is the correlation of recorded data with the state of the world. The pixels of a regular photo are highly correlated with reality - they may have been subject to some analog and digital transformation, and of course quantization, but there's a straightforward and reversible (with some loss of fidelity) function mapping pixels to the photographed event. Computational photography, in contrast, decorrelates pixels from reality, and discards source measurements, leaving you with a recording of something that never happened, but is sort of similar to the thing that did.
I elaborated on this elsewhere in the thread, so let me instead point at another way of noticing the difference. Photogrammetry is the science and technology of recovering the information about reality from photos, and it works because pixels of regular photos are highly correlated with reality. Apply the same techniques to images made via computational photography, and the degree of uncertainty and fidelity loss will reflect the degree to which the computational photos are AI interpretations/bullshit.
What captures an image is an imaging surface; traditionally a chemical emulsion on a piece of film, now a complex array of digital sensors.
This imaging surface is of human design, it therefore images what its designers designed it to image. But don't forget that it is a sampling of reality; by definition always partial, and biased (biased to the 400~700 nm range, for starters).
This does not matter in any way. What matters is that, what comes out on the other end of filtering and bias, is highly correlated with what came in, and carries information about the imaged phenomenon.
This is what both analog films and digital sensors were designed for. The captured information is then preserved through most forms of post-processing, also by design. Computational photography, in contrast, is destroying that information, for the sake of creating something that "looks better".
Well, the only thing we can say is that we can eliminate Apple phones and their software from our enquiries!
Things like lens flares don’t exist either.
Eyes do have lots of artefacts, your brain fills in the gaps, like the blind spot [1] It's not much more different than computational photography, really.
Thats irrelevant, radar reflects reality, photoshop doesn't. Even children understand this distiction.
What photo is admissible in court?
Just look at how bad fingerprints are accepted
[0] https://www.theguardian.com/us-news/2022/apr/28/forensics-bi...
[1] https://features.propublica.org/blood-spatter/blood-spatter-...
[2] https://www.currentaffairs.org/2023/01/we-need-to-get-junk-s...
It’s really hard to imagine a scenario where two exposures a fraction of a second apart could be stitched together to tell a completely false story, but maybe it exists. In that case, I suspect the lawyers would be all over this explanation to try to dismiss the evidence.
That photo was critical for the defense getting him to say that.
There were also concerns over wether or not zooming in on an iPad should be allowed in that case--like if a red pixel next to a blue could create a purple one, etc.
[0] https://www.denverpost.com/2021/11/08/shooting-victim-says-h...
This is perfectly analogous to TFA--notice that the woman has enough time to move her arms into very different positions in the same composited moment.
Before, forensic experts could decide if an image had been altered in photoshop, but I guess the only sane conclusion now is that anything taken with an iphone is fake and untrustworthy.
There's a reason professional photogs use the mulitiple snaps mode for non-sports. It used to be a lot of work in post, but a lot of apps have made that easier to now it's a built in feature of our phones.
And almost no one cares, btw.
And what do you even propose? A mandatory “picture might not represent reality” watermark? Because the way I see it, you either take the computational features away from people, and prevent them from taking an entire class of pictures, or you add a big fat warning somewhere that no one will read anyway, or you keep things the way they are. Which one of these is the ethical choice?
If cameras don't give you the option for "reality" you're just left with whatever they choose for you no matter how many pictures you take.
Does not capture the color of every pixel, and merely infers it from the surrounding ones. Is usually viewed on a sRGB screen with shitty contrast and a dynamic range significantly smaller than the one of the camera, which is significantly smaller than the one of human eyes, which is still significantly smaller than what we encounter in the real world. Does not capture a particular moment in time, but a variable length period that’s also shifted between the top part of the image and the bottom part (a couple of ms for mechanical shutter, tens or hundreds of ms for electronic). Has no idea about the white balance of the scene. Has no idea about the absolute brightness of the scene. Usually has significant perspective distortion. Usually has most of the scene out-of focus and thus misrepresents reality (buildings aren’t built and people aren’t born out of focus).
Nevertheless, what we are seeing with the post-process of the cameras described in the OP cross the threshold of what until this moment in the history is considered natural "photograph" that can be obtained analogically. Those are manipulated images, undetectable for the naked eye, that are incorporating elements, those are a "composition" that can not be obtained analogically. those images are fakes, lies.
and by the read in the comments it seems the user can not even disable such composed interpretation.
[0] https://www.theverge.com/2013/8/6/4594482/xerox-copiers-rand...
Still got room for library bugs! (wasn't the Samsung moon pictures sort of more like the xerox one?)
Photography refers to the capturing of the reflected light to create an image, which then corresponds to how the scene actually was.
Bringing out detail in shadows being called "computational photography": that I could swallow.
So it's a pano doing normal pano things, but it's just surprising in context. If you have not yet tried it, you can have friends jump in and out of the frame as you slowly move across a scene and have silly photos where they appear more than once!
Edit: typo
> A U.K. comedian and actor named Tessa Coates
Why is it always some kind of celebrity that "discovers" stuff like this? Was it luck (yet again), or are those extreme failure modes of computational photography already somewhat known, just too nerdy to report on until they can be attached to a public person?
I have used a lumix with 8X optical zoom (for a very long time) that is smaller than most smartphones (albeit thicker, thanks to the fetish for ultra-thin phones), has a removable SD card, and doesn't do processing beyond what it normally takes to make jpegs. Newer ones might have raw, I haven't checked. It has outlived many phones, has a removable battery, doesn't have a single f/1.8 aperture, and has a macro mode that doesn't require you to be 1/4 inch from the subject. If it's lost or stolen, I haven't handed over all my financial and personal data.
What will they think of next?
This has nothing to do with "Live Photos" on iOS to be clear.
Update December 2, 11:05 Updated with details about the photo being shot in Panoramic mode, which is why the bride-to-be had time to change position between the shots found in the mirror.
I still use my phone camera, but man do I appreciate that all my Sony a7 may do is interpret the sensor using a colour profile and that's it.
The poses would be at least a few seconds apart which rules out anything in Apple's computational photography pipeline e.g. focus and dynamic range stacking. At least based on what they have communicated to date.
We know Google demonstrated an AI model that was capable of selecting human parts from multiple photos but that was a showcase feature not something quietly added.
Which is well under the 2 seconds computational photography clip.
Live Photos was not enabled and even still doesn't work like this.
Are those 3 poses she's doing, or did she have her hands clasped, then performed some gesture where she dropped her left arm then right arm?
The right definitely looks like a deliberate pose, at first glance the middle does too, but the left doesn't at all, it looks like a gesture. The splay of the hands among other things indicate this to me.
I think the middle isn't a pose either, just a transition. I think only the right is a pose, and it goes Right > Middle > Left in time. Her hand is splayed in the middle like it is in the left.
And the latency for taking a standard photo is not 2 seconds.
Live Photo means storing the video in the image file. Turning it off doesn’t mean the video is no longer used, just that it isn’t stored.
> Coates was moving when the photo was taken, so when the shutter was pressed, many differing images were captured in that instant.
> Apple's algorithm stitches the photos together, choosing the best versions for saturation, contrast, detail, and lack of blur.
Live Photo is a feature where it captures a couple of seconds of video before/after the photo is taken. From the article that feature was not enabled.
The computational pipeline is where you press the shutter and it blends a few frames together in order to do focus stacking, HDR etc. Based on what I have tried with my iPhone it is doing this in < 100ms which is not enough time to produce these sort of artefacts.
I don't think photography from phone can actually be trusted to be a faithful representation of reality as it is not a purely mechanical series of actions. This will be even worse if ai is involved in the process.
I certainly think that there is an argument for photographic evidence to be inadmissible in court.