Computational photography from selfies to black holes
vas3k.com
vas3k.com
For implementing the system on an embedded device e.g. toy drone, raspberry pi etc., the main data structure you want is a k-d tree together with some sort of evergreen star chart (it doesn't have to be extremely evergreen, current astronomy libraries can easily predict orbits for a couple decades without significant skew/deviation unless you are are aiming for centimeter level geolocation accuracy).
For the hardware you can either use existing consumer-grade stuff followed by a ton of image processing with ML as suggested above or you can use a industrial grade tracker which easily exceeds 4 figures.
https://blog.satsearch.co/2019-11-26-star-trackers-the-cutti...
It's a pretty fun weekend project. Here are some links to get started:
https://github.com/mrhooray/kdtree-rs https://github.com/astronexus/HYG-Database/blob/master/READM...
Instead of jacking up your truck, add celestial nav to it. Nothing screams freedom and independence more than cutting dependency on state-funded satellite systems. Caveats: needs more signal processing during daytime, fallback to inertia navigation when it is cloudy.
> In fact, that's how Live Photo implemented in iPhones, and HTC had it back in 2013 under a strange name Zoe.
A reference to zoetropes, which were arguably one of the first "movies". https://en.wikipedia.org/wiki/Zoetrope
> To solve the problem, Google announced a different approach to HDR in a Nexus smartphone back to 2013. It was using time stacking.
I don't think time stacking is the appropriate term to use here, as standard HDR is also doing "time stacking", in that it takes multiple photos with different exposures across a small interval of time. Maybe "Fixed exposure fusion"?
I think there's a lot more to be done in computational photography, in research and engineering. In fusion of images from multiple cameras, we're barely scratching the surface. Exciting times ahead!
As a result I started using my DSLR again, and I rediscovered how beautiful the photos were. They also print nicer.
This fall I finally replaced the busted phone with an iPhone Pro. The camera on this thing is great, but the computational photography enhancements are particularly nice.
But my DSLR still smokes it.
There’s the old maxim of “the best camera is the one you have on you.” I’m happy I went with the pro, and in isolation, it’s an amazing all around snapshot camera. Occasionally I get lucky with some stunning shot.
But the DSLR just wins hands down when it comes to shooting something like the foliage of a Japanese maple. The bokeh is beautiful straight out of the camera. The iPhone still struggles, and there’s a ton of matte noise around edges that needs to be cleaned up. It’s much better at things it was designed for, like portraits.
So, anecdotally, for me at least, it’s a mixed bag. I love both cameras for different reasons but ultimately the DSLR still has the edge on quality. But the iPhone is always there, and has some tricks (especially low light) that the DSLR can’t compete with.
For the foreseeable future, I don’t see my phone fully replacing my DSLR.
In extreme cases, it even not "photo" as some information about photons recieved by some optic system with noise reduction afterwards. Not, it just pictures, based on recognised faces, objects and stars. And I don't know why, but I feel panifully bad about it. It is not approximation of world-how-it-is, but some expectation about world-how-people-want-it. It can recognise constellation based on few stars, and will draw nice picture of great stary sky, but will delete starlink sattelite, meteora or supernova as some unexpected noise.
Unprocessed camera modes will continue to exist for people who want that. Maybe with some built-in digital signature in case it is used as proof.
Photography, and earlier that that, drawing, has always been world-how-people-want-it. Accuracy is just one of the things you may want.
Yes, but also you can't leave a digital sensor collecting for two hours, the pixels start saturating and the noise builds up- not like film, which can do truly long exposures.
Some things were never easy.
Digital sensors obey the reciprocity law to a much greater extent than film: https://photo.stackexchange.com/questions/37241/does-recipro.... The problems the OP mentions are not related to reciprocity failure.
So the really interesting long focal lenght cameras are the all-in-one superzoom cameras like the canon sx70hs and nikon p1000.
They accomplish the high magnification by using a sort-of-good lens with a sort-of-good small sensor, achieving up to 3000mm "equivalent" zoom.
Unfortunately, the "pro-sumer" design gives you an electronic viewfinder and slower less accurate focus and all kinds of other non-dslr mediocrity.
sigh.
The first camera that I use that has started being ok is the pixel phone cameras using night sight to improve overall resolution and detail.
One interesting thing about Olympus cameras, that they do have a lot of computational photography things described in the article:
- there is a "Live Composite" mode, which allows you to get those star trails, to do light painting, etc.
- there is build-in focus stacking, very useful for macro photography
- there is hi-res mode implemented with pixel (or rather sensor) shifting
- there is IBIS and digital image stabilization.
The form factor make the smaller units far more difficult to hold properly while shooting, lack the optical viewfinder in most cases, etc.
The only thing a person needs for Star trails is a tripod and the sky. Stars start to "trail" generally anything over 500/effective focal length in exposure time in seconds.
Built in focus stacking does sound useful if it works well and macro shots are of interest.
Hi res modes are fine but it can never make your pixels bigger - bigger pixels of the same generation more sensitive to light so less noise at higher isos.
I have canon glass so tried one of their mirrorless cams and found its Ergonomics completely different to DSLRs and personally not in a good way.
The only way would be a completely fake ML based manipulation of the image to distort the image based on object/scene detection.
> The beauty of the situation is that we can't even imagine today what it's going to be.
Optical phased arrays of nanoantennas?
Let's take android - the snapdragon 855 is a very standard flagship CPU and is on tons of phones (including the 450$ Xiaomi K20 Pro). Why isnt there a computational photography app that works on these phones ?
Why is Pixel 4 - which uses the same 855 chip - the only one that has this software. Is it patent encumbered ? or is there some massive dataset deep learning stuff involved.
I'm surprised that there isnt a startup that is building these apps out there.
The image processing speeds up the processing - I admit to it. But we have the 855 flagships which are probably a few seconds behind. This is ok with us.
I'm asking if anyone can hit the quality even given 5x more processing time .
Cause all the tricks used on smartphones still work if you have a better lens and sensor. They will still yield these improvements. It's just gonna take a while for the manufacturers to catch up. Especially on the DSLR end, because of slow shutters and massive momentum.
Let’s be respectful to the original author, I spent couple of months writing it:(
What a shame! They've completely rip-off your article and "thankful" for that!
My name is Kate and I am Head of Content at Let's Enhance.
We are very thankful for the given material, because it's highly related to us.
And I want to clarify this unpleasant situation:
I've directly talked with the author of the article - Vas3k and asked for the permission to publish this awesome material. I can attach the screenshots of our conversation.
All the copyrights are reserved. We mentioned you and your blog as the original source. What's more, we've saved all the links in the article that refer to your blog.
So, all the accuses are unfounded.
>We mentioned you and your blog as the original source.
Just because you reference a work it doesn't mean it's not plagiarism when you straight up copy most of the content.
It is more appropriate to a preamble stating that the following images and text are directly copied from the original author.
Saying we are thankful for "given materials" is too vague (and probably offensive to the original author) when all the images, text, and even structure of the article is a copy.
Oh, and stealing ad revenue from the original creator.
> The article was originally published by Vas3k in https://vas3k.com/ on 30.06.2019 Let's Enhance team is very thankful for given materials.
Granted I still agree it's a blatant copy/paste and I also vote that dang replaces the link with the original.
And even a mind-bending (or optic?) glimpse into the future.
Thank you for writing it.
(Source: I'm in the VR / plenoptic space, knew a bunch of people at Lytro, some of whom are now at Google. Timelines and facts do not match this article's assertions.)
Phase detection is not the same thing at plenoptic; the article implies that phase detection operates in the same way as a microlens array. It doesn't; microlens arrays for plenoptic imaging have many pixels underneath, and perform different operations on them. This line:
"With only two pixels in one, there's still enough to calculate a fair optical depth of field map without having a second camera like everyone else."
This is categorically false. You need dozens of pixels underneath a microlens array to do digital refocusing; phase detection is not remotely the same thing.
Later, the author conflates microlens-based refocusing with camera-array based plenoptic imaging. These are wildly different disciplines; the work done in VR does not include microlens arrays anywhere. Not any. But they do include large arrays of cameras (see volumetric work from Microsoft HoloCapture, Intel's capture studio in Manhattan Beach, and my own company Visby's work — cameras at Radiant Images in LA... all arrays of cameras, no microlenses anywhere).
"Apparently, if you take only one central pixel from each cluster and build the image only from them, it won't be any different from one taken with a standard camera." This is false; taking the central pixel will give you an aliased image (or an image as though taken with a very, very high f-number lens, if the pixels correspond to the same zone of the aperture). To make an image as though you didn't use a microlens array, you sum all the pixels under the microlens to produce a macropixel. This is why light field cameras have lower effective resolution than traditional cameras. The remainder of the speculation about 'sneaky plenoptic JPEGs' has no basis in light field imaging; refocusing is not achieved by throwing away (or binning) pixels, it's done by summing different sets of pixels. Lytro founder Ren Ng's PhD thesis was about a way to do it faster by using the Fourier Slice Theorem. (Beautiful paper, if you have time to read it.)
Google did not "buy and kill Lytro." Lytro's assets were sold and Google bought some of them, but none of that tech made it into the Pixel line of cameras, period. The Google portrait mode is done using phase detect pixels as a cue for segmentation, as discussed in the article this blog links to. Nothing plenoptic required, and certainly no Lytro tech there. The blog post implies that the Pixel is using Lytro-like techniques rather than phase detect pixels to hint at segmentation for synthetic blur.
I have never heard of plenoptic cameras being used for image stabilization. In my personal experience (5 years of stabilizing cinema cameras for two different companies), virtually all stabilization issues derive from rotational blur, not translational movement. Your ability to stabilize translational movement would be limited to the aperture size, and translations of that magnitude have vanishingly small effects on the image unless the objects are very, very close to the lens (like macro). I have no idea where the blogger came up with this idea but there is no reference provided and I have never heard of it (nor would it work, for the above reasons).
The section "Fighting the Bayer Filter" miscasts the challenges of a Bayer pattern under a microlens; naively summing adjacent pixels under other microlenses would blur an image. You do get a boost to dynamic range from the synthesized images — you're binning a bunch of pixels to create your image, so you reduce noise as 1/root(n) in the number of samples. That could lead to better color in dark, noisy areas, but it will not increase the gamut of your sensor.
It goes on... but since this article was already off the front page by the time I posed my comments, this is probably enough.
(As an aside, Jacques — I really enjoy your writing!)