Lensless camera creates 3D images from single exposure
optica.org
optica.org
With more modern sensors you might be able to fix the resolution issue as I recall it being around 2010 or so.
The overview of the 48 megapixel camera on the iPhone 14 talked chipperly about the fact that the user-land resolution is already decimated (well, quadrimated?) 4 to 1 to 12mp images, with the subtext "but those are highly determined optimized pixels!"
Just now we're starting to see 40Mp APS-C sensors, and people are debating if it's worth it to do that on a sensor that's like 10x larger than the biggest iphone sensor.
The Rayleigh criterion is admittedly pretty lax from the point of view of imaging. People often talk about digital cameras being "diffraction limited" if the diameter of the Airy disc is larger than the dimension of a pixel (which is clearly the case for the iPhone). However, AFAIK, there is no strict limit here. It's something of an open question how good smart sharpening algorithms can get – especially when they're allowed to 'cheat' by using knowledge of the statistical properties of typical real world scenes.
The pixel size for a 48MP APS-C sensor is about 2900 nanometers, so it's not 10x bigger (even if you compare area).
True, but you're strictly limited by what I assume are microscopic gate capacitances on pixels that small. The maximum number of electrons in an iphone pixel are probably less than 10k, leading to less than 7 bits of maximum theoretical entropy per px from shot noise limits. This is going to sharply curtail (no pun intended) your ability to "back out" airy disk overlap.
1: https://twitter.com/karenxcheng/status/1564636065410953217
Did Lytro have a sensor that could actually measure both the position and direction of an incident light ray ?
Reading the article, I don't at all see what the difference is between the Lytro camera and what's described there - although I'm not an optics expert.
The average person, or probably even the average VC, isn't going to read a pitch about being able to take photos without having to adjust lens focus and say "gee, with that technology you could build a 3D model as easily as taking a photo".
Applying the same principle in reverse, you can also create a 3D display by adding a microlens array on top of an image. If you use the same microlens array to capture the image as to display it (swapping out the sensor with a display, or a camera film with its developed photograph) then optical physics does all the necessary image processing for you.
The idea goes back at least as far as Gabriel Lippmann in the 1900s: https://en.wikipedia.org/wiki/Lenticular_printing#Gabriel_Li...
It's just a property of light field cameras.
Technically, other cameras can do it to some extent too. If there's some opaque object with another object behind it,
1. If you focus on the front object, you'll see the object behind it--blurred.
2. If you focus on the back object, you'll be able to see the back object behind the blur of the front object.
With a light field camera, you can essentially "cut away" the blurry parts and get them in focus. You're not able to see anything new, you're just able to see things in the original image more clearly.
So they built a camera with a microlens array and refocussing... Like this one, which has been commercially available for 10+ years? https://raytrix.de/products/
They tried to market to sports and nature photographers since it eliminated the need to focus before taking a picture. Thus they would have a better chance of capturing the perfect image at exactly the right instant. Unfortunately, after doing the software focus post-processing the images always looked a little soft. And the autofocus features on DSLRs got faster and more accurate. So there was no market.
This isn't technically correct. Everything was in focus, because the aperture was so small. The software would add an artificial blur, using the 3d models generated.
So, you chose how to artificially blur the image, after taking the exposure.
That's... not how you make a new product popular and make tons of money. That's how you make a new product a niche product. That's how you kill a product.
Vendor lock-in is for when you have already acquired the mindshare. Users hate vendor lock-in.
To start out having vendor lock-in baked in you'll have to have such an incredibly amazing and useful new product -one that's in its own category- that users will disregard the lock-in for having no choice. The Lytro camera is not such an amazing product.
Mindshare is extremely valuable. Vendor lock-in is a tool for when your competition is rising, but it's very risky, as it may close your product to new customers while making your existing customers want to jump ship as soon as practicable. Vendor lock-in is very risky not just for the customer but also for the vendor.
Check out this archived post from when they took down their photo sharing site: https://web.archive.org/web/20180327235711/https://lytro.wuf...
If they actually wanted to build a popular product with significant longevity, they could have done FAR more to enable people to build on their file format etc.
This "they didn't want to pursue full vendor lock-in at every step of their company; they had no choice" apologism just doesn't stand up to any scrutiny.
More likely, they made a bid to capture a greater share of the value, at the cost of limiting the size of their market, and it predictably failed. Or maybe they were prioritizing some kind of acquisition, over building a company with real longevity.
I don't really know anything about the company besides a short skim through Google, but it just doesn't seem remotely plausible to me that they really truly wanted to make this accessible to more people and cultivate a larger market and ecosystem, but somehow had literally no choice in pursuing their vendor lock-in strategy.
They chose to bet on control over mindshare, and this had predictable consequences.
Also the device itself looked nice, but it was fairly hard to use and the view finder screen was a fairly bad lcd even at the time.
The fact that they weren’t giving away decoders at the time is pretty much a given, especially given how much emphasis they put on the “algorithms” they used.
Now the resolution is less a problem with plenoptic 2.0 cameras, in [0] they extract full HD images
First sentence: "Researchers have developed a camera that uses a thin microlens array..."
“ To the best of our knowledge, we are the first to demonstrate deep learning data-driven 3D photorealistic reconstruction without system calibration and initializations, and we are the first to demonstrate imaging objects behind obstacles using lensless imagers.”
That’s their actual claim, worth examining.
Also AFAICT their camera has no large traditional lens at all, whereas previous work — ie lytro — put a traditional lens in front of their micro lens array.
Opinions are opinions, peronally I would consider it multi-lens or poly-lens.
A regular lens manipulates the light so that each sensor corresponds to one pixel, mimicking the way a piece of film would produce a color at each spot.
With a light field camera the actual image has to be constructed computationally afterwards. In effect, the software computes the effect a lens would have had on the light -- meaning you can set the focus later.
There is a lens-like apparatus built into each sensor to allow it to gather the types of data it needs but it's not at all like a conventional camera lens. You couldn't put on a conventional camera lens, and none of the things that a photographer uses a "lens" for are necesary. "Lensless" seems as good a word as any for that: it accurately conveys a camera that you do not and cannot focus, and has no object corresponding to the lens.
I don't think this is technically correct. There are many focused images being produced: https://www.youtube.com/watch?v=rEMP3XEgnws
> but it's not at all like a conventional camera lens
The physics is exactly the same as a conventional lens, because each micro lens is conventional lenses.
It's all conventional, up to the point of having to separate all of the real, focused, images to make one clear image.
For proof, if you block all the micro lenses, except one, you would see a boring, focused, real image projected onto the sensor. In fact, the original light field cameras, that you could buy off the shelf, had a single, separate, image per microlens. Looking at the image in this press release, it looks the same.
At this point we could connect each sensor to its own circuitry and storage and then say it's a cameraless photo altogether :)
I bought a Lytro camera that was doing this possibly more than a decade ago. It was a “consumer” device, but it was pretty useless (though kind of pretty to look at?).
Essentially you lose huge amounts of 2d resolution on the sensor so the resultant picture were both very low res and also huge. Coupled with no standard format to be available, I vaguely recall that the only way to “share” the images was via a Facebook applet thing.
I assume a decade of work means they can process the data faster and get better resolution, but the tech itself is far from new.
Also you can title an article “this camera has no lens” and then start with “it has a microarray of thousands of lenses” :D
The Lytro camera was on the market in 2014.
It's weird to think the real computation is physics, and machines or optical systems are nested abstractions hosted on that.
I guess one person's low-level / "metal" / "wire" is always someone else's API.