Cameras and Lenses
ciechanow.ski
ciechanow.ski
https://ciechanow.ski/lights-and-shadows/
https://ciechanow.ski/tesseract/
Etc.
Check out the whole blog! It's amazing.
These are some of the most intuitive explanations coupled with the slickest animation / demo work I've ever seen. All of that put together into crystal clear educational material is such a rare gem and requires incredible talent.
Impressive as hell.
I hope someone is paying them to do this full time. I'd pay for this.
> I find it very inspiring that while we can’t physically experience a four dimensional space, with just a bit of ingenuity we can easily simulate how a tesseract and its shadow would look in our day-to-day world.
> You may find math’s indifference to the limitations of our human perception quite cruel, but I think it’s liberating. Reflecting on higher dimensions is transcendent – it removes the shackles of the physical world and allows us to explore the realms we’ll never encounter.
My one minor nit-pick would be introducing the wave nature of light so early--to motivate Snell's law and total internal reflection, but without discussing diffraction until well later and only in passing.
Snell's law can be motivated from a purely classical particle-based model as:
1. Light travels slower through media other than vacuum (This is mentioned, but hand-waved as boundary conditions, which probably isn't meaningful to the intended audience). 2. Light takes the shortest (in time) path between two points. (Light... finds a way).
Either way, the audience might find the obvious "why" for the first point more satisfactorily addressed with a short sidebar on permittivity and permeability of materials (their electrical properties, effectively the capacitance and inductance "density").
That can also lead to an interesting discussion of why conductors are usually not optically transparent and insulators are.
It just needs a diagram showing the waves refracting, and the geometry of the wave fronts. (Pretty annoying that they never explained refraction like that in school either - Snell's law was just a given when it's actually really trivial to derive from first principles.)
I'd highly recommend it to anyone who is seriously interested in the underlying 'mechanics' of imaging.
(Incidentally, anyone whose has read my many anti-JavaScript raves, then I'd only say this is exactly how JavaScript ought to be used.)
Let me explain:
When you have two holes (apertures), their images would likely overlap (depending on the arrangement).
Imagine you took two photos from a few millimetres apart to the left and to the right, then you superimposed them. Some areas would look ok (as both views saw the same distant object) and some would be less ok (of objects that are closer).
If you have a DSLR take the lens off and put a sheet with holes in front. Just try it.
Now as you add more holes you get more and more overlapping images on the same area of the sensor and eventually end up with a blur.
This blur is the same blur you see with a single large hole. It's just that with a large hole you had infinitely many overlaps!
Even cooler, if you happen to arrange the holes in a specific pattern you could capture images with different combinations of perspectives from different holes and you may even undo the overlaps. This is called coded aperture imaging:
https://www.paulcarlisle.net/codedaperture/
This doesn't just solve the biggest problem (limited light) of a single hole, but also captures depth information and you can use it for 3d reconstruction, refocusing etc.
One final bit, with a warning of a deep rabbit hole:
That "infinitely many overlaps" I was talking about happens with lenses too and is essentially a convolution where you convolve the image with itself (actually many different perspectives of itself if I am correct). Which is just the Fourier transform.
That statement is a bit muddled, let me unpack it. The infinitely many overlaps thing can be expressed mathematically as the convolution of the image with a function that's one where the aperture is open, and zero where it's not. The thing about the Fourier transform is actually related to a different phenomenon. When the slit is really small, you start getting diffraction effects. The diffraction bands are approximately the Fourier transform of the slit function. However that is not significant unless the slit is extraordinarily tiny.
Is this correct from a physics standpoint? Feynman had a lot to say about light in his book (QED https://www.amazon.com/QED-Strange-Theory-Light-Matter/dp/06...) and never framed it this way. At one point he remarked that the explanations in the book were related to diffraction, and the explanation there was very different from the Fourier transform.
Physical optics takes into account the wave nature of light. This becomes important when the size of the lens becomes small (eg pinholes) ... there's diffraction around edges and pixels receive contributions from many points in space.
Geometric optics lets you model using ray-tracing, reflections, and Snell's law refraction.
Physical optics uses tools such as Fourier transforms, convolutions, and sinc functions.
Understanding a simple lens system? Geometric optics is your friend. Building an astronomical telescope? Check into physical optics.
That’s crazy cool! Got any good links for reading about how to do those things that way; 3d reconstruct and refocus?
Stanford has a class on computational imaging: http://stanford.edu/class/ee367/
I think these slides should give you a pointer to enough papers to fill a lifetime: https://web.stanford.edu/class/ee367/slides/lecture9.pdf
Georgia Tech also has a class, that's on youtube and Udacity: https://www.udacity.com/course/computational-photography--ud...
I cannot imagine the effort that goes into writing and coding a post like this.
https://ciechanow.ski/js/base.js https://ciechanow.ski/js/lenses.js
Many people also debayer their cameras themselves, especially in astro photography [1]
[0] https://en.wikipedia.org/wiki/Foveon_X3_sensor
[1] https://stargazerslounge.com/topic/166334-debayering-a-dslrs...
source: I've done this mod on a couple cameras and wouldn't recommend it for performance gains alone. It's a fun project, not much more.
Data values captured by a modern camera far exceed the ranges that can be reproduced by output media such as paper or screens—meaning that the only way[0] to obtain an image we can communicate to someone else (or future ourselves) as a static artifact[0] is by throwing out data and conforming values to ranges that fit in the output color space, converting scene-referred (“raw”) data to output-referred.
This is where subjective perception comes in: how we perceive colors and shapes in a given scene depends a lot on what we had seen prior to this scene, our general mood, and various other aspects of our mind state. It’s only by taking control of processing scene-referred data that we can use the full range of captured values to try to convey most convincingly, within the constraints of the output space, our subjective perception of the scene around the time we pressed the shutter trigger.
(Naturally, further down this rabbit hole come the questions about e.g. what our conscious perception—but not the camera—was blind to in the scene, and eventually about the nature of reality itself, at which point one may feel compelled to give up and go into painting instead.)
[0] This would be quite niche, but I wonder if we could develop tools for exploring raw data at viewing stage, allowing the audience to effortlessly adjust their perception of the scene (even if within ranges specified by the photographer). Such exploration would require significant computing powers, but we’ll probably be there in a couple of years.
The tool interface needs to be simplified to make it a better fit for the use-case you present but I don't see computing power as a bottleneck.
Of course they'd still be limited by the dynamic range of the camera. This can also be resolved by calculating irradiance map based on multiple RAW images taken with different exposure times.
Apart from performance, another crucial thing is that the viewer must not have to think about technical aspects (like exposure, color profile, etc.), as you noted, so the GUI would have to be radically different.
I am envisioning producers bundling N processing profiles with their “digital negative”, and software that somehow allows the user to fluidly explore the perception of the scene by interpolating inside an (N-1)-dimensional space bounded by parameters in those profiles with really low latency.
It can't be "bokeh" because that's just an everyday word that means blur; for instance "mae no shashin ga boketeru kara, keshimashou ka" (the previous picture is blurry, so shall I erase it?)
Also, since that term came into use in around 1997, if I could have a conversation with an English-speaking photographer via time telephone to 1990, what would be the wording used?
I'm in awe.
A great way to visualize this, is to take a look at the sharpness of the nifty fifty canon lens
https://www.imaging-resource.com/lenses/canon/ef-50mm-f1.8-s...
Click into the heat map https://www.imaging-resource.com/lenses/canon/ef-50mm-f1.8-s...
Move the aperture from 1.8 to 2.8, and note how the sharpness increases, starting in the center of the lens.
Then at about 16, diffraction kicks in and things start getting less sharp again.
Things get even wilder when you introduce variable zoom and different camera bodies.
It's a great way to visualize the good/fast/cheap pick two tradeoff's of lenses.
I love this writer / developer
I've been working with computer vision since university, but now days work underwater. I guess I should have a pretty good understanding of this stuff, but I still occasionally get caught out by optics.
This is the best possible pre-reading for anyone wanting to jump into the field.
One interesting thing I learned recently is how they make cheap plastic lenses, and the fact that plastic lenses were invented by an Australian. They are created by choosing a polymer with good optical properties and then thermoforming it by carefully applying air pressure to raise a rounded shape to a dome after clamping the edges, then holding the air pressure constant until it has cooled. Because nothing actually touches the lens portion, optical properties are preserved.
And then, no further polishing is necessary? I'm fascinated to learn more if you have a source to read more from!
> In real camera lenses an adjustable aperture is often constructed from a set of overlapping blades that constitute an iris.
If you naively create 6 blades and rotate them around the pivot (the tiny screw), they will never overlap EACH OTHER. One will end up at the very top, and one at the very bottom.
If you know how this was created, please let me know.
I'm guessing they drew overlapped part of the last blade twice
Here’s a metaphor. Let’s say you have 10 glasses, with 10 different amounts of water in them. When you hit them with a spoon, each glass makes a different musical note: 10 different notes.
If you pour the water from each glass individually into 10 new glasses, you preserve those 10 notes.
If you combine every two glasses into one, now you have 5 glasses, and only 5 notes. It’s the same total amount of water but you lost information.
Look up circle of confusion, Rayleigh resolution criterion and sparrows resolution criterion for the technical details of resolution.
The entire object doesn't get collapsed to a single point. Rather, a single point of the object radiates light in all directions. A lens then captures a fraction of that radiation and collapses it back down to a single point. Then we iterate over each point in the scene with a "for each".
- water: when waves go from, say, deeper to shallower water, their speed changes. If they are moving at an angle, refraction is seen. This can be explained by drawing wavefronts. The wavefront doesn't hit the shallower water all at the same time, and so does not slow down all at once, but gradually.
- wheels: when rolling axle with two wheels goes from pavement to grass on an angle, its direction will bend direction, due to the change in speed, which is experienced by one wheel ahead of the other.
It really helps building an intuitive foundation of these phenomena.
With such intuitive foundation, one has confidence to look into the math behind the curtain, even if this math gets complex, you will still feel confident.
IMO, this kind of explanations is the correct way to start teaching anything.
Umm well, not really.
The sensitivity of a digital CCD or CMOS sensor never changes; it has one 'native ISO' equivalent sensitivity and that's all. It creates the same output signal in response to a photon every time.
Varying the pseudo-ISO simply varies the gain of the downstream post-readout amplifier. That's where noise is introduced.
Sometimes it's better to take a deliberately underexposed photo and tweak regions of it in post to achieve the desired exposure instead of introducing global noise from in-camera amplification.
With an image sensor you get to decide between high sensitivity and high capacity (how much light it can detect before saturating). So a highly sensitive sensor can handle less light before clipping (which makes sense).
This trade-off can be tweaked by changing the size of the photodiode (bigger diode -> more sensitive), but also electrically - simply add capacitance in parallel to the photodiode. This requires more energy, i.e. more photons, to change the voltage over the photodiode, which is what the readout measures.
Dual native ISO sensors simply add another transistor (as a switch) in series to extra capacitance; this allows them to switch between low-capacitance (high sensitivity, but saturates faster) and high-capacitance (lower sensitivity, can handle more light).
Edit: Haven't watched the whole video but his core explanation in there seems to be that Dual ISO means the sensors has a PGA (programmable gain amplifier) -- almost all sensors use that approach, and that's what is controlled by the ISO setting in them.
That is definitely not my experience for Canon's in low light situations. Here [1] is an example shot at ISO 100 but pushed by 5EV to match the brightness of the same scene shot at ISO 3200. The noise is much more tolerable for the latter.
[1] https://imgur.com/a/Ca9ccbK
P.S.: it's raw, no in-camera JPEG denoising in place.
Color filters on the sensor split the light into three wavelength ranges - red, green, blue. Then photosites measure intensity of light, which means that sensor only knows that the incoming photon is "roughly green", but doesn't actually recognize its precise wavelength.
So e.g. when the light is pure orange, it falls down into red wavelength range and is counted as red.
Based on this I would expect that cameras would often produce pretty incorrect colors, but they are usually pretty good (after correcting for stuff like white balance).
It is important to note, that neither the sensor pixels nor your eyes have a complete separate reaction to a wavelength. The sensitive area strongly overlaps. So for hues of green which have rather long wavelengths, you get some reaction on the red pixels, which gets stronger as you move towards orange, where both red and green pixels detect until it gets more and more red and less green. The exact absorption curve of the sensore color filters matters here, that is one reason, different manufacturers have slightly different color rendition. On top of that is calibration, when converting the raw image into a proper RGB-image, one can further balance the response. For that, the color calibration targets are used, which have like 24 patches of different colors. Taking a phone of this target, the calibration software can both calibrate for the light illuminating the target as well as the color response of your camera.
A common reason for red-green colorblindness is that the affected persons have the sensitivity between the red and green colors overlapping too strongly, so they loose the ability to differentiate. A green creates almost as strong a signal in the "red" cells. A way to improve the color vision for those people are glasses which increase that separation by absorbing the frequencies between the red and green colors.
Well, depending how you think "directly" but you can get pretty far with spectrometer, i.e. device that splits light and measures the intensity spatially to collect a spectrum. It's not impossible thought to build camera based on that principle, just need to sample the light in an array to make pixels
Different sensors/cameras have different filters, and combined with the manufacturer specific post-processing this gives different cameras/manufacturers a different color rendition and feel.
'Pure' orange light isn't red. It has a wavelength of 590–620 nm. Red is 625–740 nm and Green is 495–570, so 'orange' is between red and green. The sensor filters each allow a range of wavelengths through. So green is triggered as well as the red filter. In RGB terms orange is 255, 127, 0, i.e. with a strong red component and a smaller green component.
White balance is computed downstream from the sensor, and is used to resolve the effect that a coloured light source creates a colour cast on objects, most noticeably on white ones. The human visual system auto-compensates for this but cameras require special processing, sometimes done using presets for different types of light (sunlight, shade, tungsten etc).
https://en.wikipedia.org/wiki/Color_vision#Physiology_of_col...
If you only work with one photon at a time, then yes, you don't know if the photon passing through the filter was red, or orange, or with smaller probability even green or blue. But when you have trillions of photons passing through, you can "see" the difference by the relative intensity of light.
Remember that at the quantum level, things don't happen deterministically; you have to consider the probability that a given outcome occurs. So the photon has a certain probability to hit the filter and get absorbed, a certain probability to pass through the entire depth of the filter, a certain probability to hit the sensor without generating an event, a certain probability to hit the sensor and initiate a chemical reaction (for film or biological eyes) or an electron cascade (for CCD sensors), a certain probability to quantum tunnel to the other side of the universe...
So getting back to your question, when pure orange photons hit a red filter, many of them will make it through the filter, but not as many as if they were pure red photons. When pure orange photons hit a green filter, some of them will make it through, but not as many as if they were pure green photons. So if your brain knows what the "white point" of a given environment is, relative to that white color it'll see a specific combination of "some red, some green" as orange. (Of course, if known-to-be-white objects are already orange--like when you put on amber ski googles--your brain will eventually adjust and recalibrate to perceive that color as something else... perception is tricky!)
The goal of a digital camera is to be sensitive to colors in the same way that your eyes are. If that tristimulus can be recorded and played back, your brain won't know the difference. The colors your monitor emits when viewing a photograph could be totally unrelated to what was in the original scene, but since your brain is just looking for a tristimulus, as long as that same tristimulus is produced, you won't be able to tell the difference.
(Fun fact -- there are colors you can see that don't correspond to a wavelength of light. There is no single wavelength of light that fully stimulates your S and L cones, but plenty of things are magenta.)
TL;DR: computerized color is basically hacking your brain, and it works pretty well!
So as long as the filters in the camera have roughly the same transmission curve as the sensitivity curve of the color receptors, all is good.
However, to animals (or hypothetical aliens) with other color receptors, the images produced by photo prints and screens would look quite weird, with colors all wrong.
Having this article back then would have been a big help! The interactive widgets are great, and it looks like they're entirely home-grown. I am really impressed with the level of detail and work that went into this post.
And the Canon Dual Pixel AF that works a bit different than the article explains https://www.canonwatch.com/dual-pixel-af-has-become-a-canon-... (but the substance is the same)
I’m a novice when it comes to anything related to web development so I’m going to go and look at how this type of this is done. Would anyone have and suggestion or recommendations for getting started?
I teach fluid mechanics with Jupyter and I use pyGEL3D for inserting 3D models but it’s far from ideal. I’ve love to do something like this.
All of the sliders should be labeled.
All in all, this is a good conceptual effort, but the details aren't strong.
I wonder if the author publishes the source code for the interactive bits somewhere (I could of course look at them from the presented JS, but still). I've been looking for a non-fan-inducing way to generate a DOF-effect, and the one in the first example is definitely good enough for my use case.
Edit: Now I've seen some of his other pages... This man is a genius.
There is a demonstration later in the article (below the 5th mention of "focal length") which does affect the lens (blue ring) in the left-hand pane. Rotate the scene so that you are viewing the camera side-on, and you will see the focal length of the lens changing when you move the slider.
I am curious, however, why we still can’t digitally reproduce bokeh. Apple is getting close. I thought LiDAR would theoretically solve that and could yield indistinguishable renders compared to analog lenses. That would be a game changer in my view and why I would like to see Apple develop a full-frame sensor coupled with their technology.
Bokeh is a kind of of space representation, similar to how you can basically "see" through hearing a sound stage of instruments separated properly when someone has a really good sound system, or how dogs have "5.1/7.1" sense of smell.
How does one encode that I have no idea.
In the world of 3D rendering (content created from scratch) very advanced & realistic bokeh effects are possible, as an example see http://lentil.xyz for the Arnold renderer.
* https://www.researchgate.net/figure/a-b-The-first-and-second...
Also, I'm excited about cameras with dual/quad pixel AF, that are kind of a hybrid between lightfield and traditional cameras. I wonder what kind of sorcery one would be able to do with the light field data in those cameras!
The thing is, these tails become apparent only when you convolve with a very very bright source -- which on a typical 12-bit level linear raw image would amount to something like 10⁵-10⁶, i.e. needing 4-8 additional bits of HDR.
Here are some useful links on the topic of flare simulation, I believe bokeh has mahy similar aspects:
https://web.archive.org/web/20200119024053/http://simonwinde...
[1] https://people.eecs.berkeley.edu/~malik/papers/debevec-malik...
I'm using Firefox (Nightly) on a Windows 10 tablet (Microsoft Surface Go).
I've tried with Chrome on Windows 10 and find that the slider works but only if I touch on the line the slider is sitting on to set a value. Attempting to tap on the round circle and pulling it to slide does not work.
I tried again on Firefox and if I very carefully tap on the line, I can set the slider. But touching the circle and sliding does not work. So, Firefox is more sensitive than Chrome when it comes to determining whether I am touching the line or not.
I presume this is due to how Windows interprets touch events and passes them on to the browser, making the sliders hard to work with in my case.