Hacker News Reader

Comment by birdman3131 | Hacker News Reader

We have had this for years. The term is Light Field Camera. There was an attempt several years back to do this with the Lytro camera. It worked but had a few major issues. For one the resolution loss was fairly substantial. There was also a fair few issues around the fact that the only way to share the photos was a cumbersome vendor lock in site iirc.

With more modern sensors you might be able to fix the resolution issue as I recall it being around 2010 or so.

View on HN

Agree, this appears to be a pure PR piece.

So they built a camera with a microlens array and refocussing... Like this one, which has been commercially available for 10+ years? https://raytrix.de/products/

I think the catch is different on this one. They seem to co-optimize the optics hardware and the processing software. They say that new manufacturing allows this kind of stuff. Which I think is cool.Still I wonder how much difference this will realistically make. In my experience (from printed electronics) new customizable manufacturing processes often also come at some cost and expose a lot of new effects, so is difficult to break even with something that keeps one side stable. Although particularly in optics new smartphone lenses I guess have proven how good you can get with such approaches.

Resolution will always be an issue since it's a trade off of dedicating a fixed area to single 2D plane or a fraction of it to multiple planes in 3D.

That's true, but it might be mainly a cost issue. You can make up for the resolution loss by using a much larger sensor; since the microlenses are very thin, it won't increase the weight or bulk of the device very much. Just the cost.

This.

The overview of the 48 megapixel camera on the iPhone 14 talked chipperly about the fact that the user-land resolution is already decimated (well, quadrimated?) 4 to 1 to 12mp images, with the subtext "but those are highly determined optimized pixels!"

The 48mp sensor on the iPhone seems like a gimmick - even with purely diffraction-limited optics, I doubt you could seriously increase optical bandwidth at that sensor size beyond, say, 10-20Mp, to say nothing of aberration limiting and shot noise constraints.

Just now we're starting to see 40Mp APS-C sensors, and people are debating if it's worth it to do that on a sensor that's like 10x larger than the biggest iphone sensor.

Some simple calculations suggest otherwise. The iPhone 14 Pro 48MP camera has a pixel size of 1220 nanometers. The diameter of the Airy disc at f1.78 for 500nm light is 2170 nanometers. Seems like Apple has designed things so that they're just on the right side of the Rayleigh criterion.

The Rayleigh criterion is admittedly pretty lax from the point of view of imaging. People often talk about digital cameras being "diffraction limited" if the diameter of the Airy disc is larger than the dimension of a pixel (which is clearly the case for the iPhone). However, AFAIK, there is no strict limit here. It's something of an open question how good smart sharpening algorithms can get – especially when they're allowed to 'cheat' by using knowledge of the statistical properties of typical real world scenes.

The pixel size for a 48MP APS-C sensor is about 2900 nanometers, so it's not 10x bigger (even if you compare area).

> It's something of an open question how good smart sharpening algorithms can get

True, but you're strictly limited by what I assume are microscopic gate capacitances on pixels that small. The maximum number of electrons in an iphone pixel are probably less than 10k, leading to less than 7 bits of maximum theoretical entropy per px from shot noise limits. This is going to sharply curtail (no pun intended) your ability to "back out" airy disk overlap.

Stacking multiple exposures should help with that.

I wonder if something like DAIN[1] could be used to take fewer planes and then interpolate. Maybe that way the tradeoff wouldn’t be as bad?

1: https://twitter.com/karenxcheng/status/1564636065410953217

> There was also a fair few issues around the fact that the only way to share the photos was a cumbersome vendor lock in site iirc.

That's... not how you make a new product popular and make tons of money. That's how you make a new product a niche product. That's how you kill a product.

Vendor lock-in is for when you have already acquired the mindshare. Users hate vendor lock-in.

To start out having vendor lock-in baked in you'll have to have such an incredibly amazing and useful new product -one that's in its own category- that users will disregard the lock-in for having no choice. The Lytro camera is not such an amazing product.

Mindshare is extremely valuable. Vendor lock-in is a tool for when your competition is rising, but it's very risky, as it may close your product to new customers while making your existing customers want to jump ship as soon as practicable. Vendor lock-in is very risky not just for the customer but also for the vendor.

It's not like they wanted to do this. They had no choice because standard photo viewers can't refocus photos. And a Lytro photo without refocusing is... just a photo, and a low-resolution one at that. So they had to build a UI for refocusing and host the shared photos themselves.

They had no choice? Really?

Check out this archived post from when they took down their photo sharing site: https://web.archive.org/web/20180327235711/https://lytro.wuf...

If they actually wanted to build a popular product with significant longevity, they could have done FAR more to enable people to build on their file format etc.

This "they didn't want to pursue full vendor lock-in at every step of their company; they had no choice" apologism just doesn't stand up to any scrutiny.

More likely, they made a bid to capture a greater share of the value, at the cost of limiting the size of their market, and it predictably failed. Or maybe they were prioritizing some kind of acquisition, over building a company with real longevity.

I don't really know anything about the company besides a short skim through Google, but it just doesn't seem remotely plausible to me that they really truly wanted to make this accessible to more people and cultivate a larger market and ecosystem, but somehow had literally no choice in pursuing their vendor lock-in strategy.

They chose to bet on control over mindshare, and this had predictable consequences.

Nah what killed Lytro was that it was very expensive, while also being low resolution, while also not really offering a feature that people really wanted. People basically just want the multi focal plane stuff that modern phones do but cheaper and at higher resolution. I don’t think people have ever really demonstrated a desire to share images that could have the focal plane changed by the viewer, but even when you could with this format the functional number of planes was actually quite small.

Also the device itself looked nice, but it was fairly hard to use and the view finder screen was a fairly bad lcd even at the time.

The fact that they weren’t giving away decoders at the time is pretty much a given, especially given how much emphasis they put on the “algorithms” they used.

The Lytro camera allowed you to choose how to focus the image after taking the exposure. So naturally the only way to do that was with their proprietary software because there was no suitable industry standard. It did then allow for exporting in standard formats like JPEG.

They tried to market to sports and nature photographers since it eliminated the need to focus before taking a picture. Thus they would have a better chance of capturing the perfect image at exactly the right instant. Unfortunately, after doing the software focus post-processing the images always looked a little soft. And the autofocus features on DSLRs got faster and more accurate. So there was no market.

> The Lytro camera allowed you to choose how to focus the image after taking the exposure.

This isn't technically correct. Everything was in focus, because the aperture was so small. The software would add an artificial blur, using the 3d models generated.

So, you chose how to artificially blur the image, after taking the exposure.

The title is misleading about the work. The real novelty here is the particular deep-learning processing for the 3D reconstruction.

Yes, this is a plenoptic camera, the main difference from current methods (e.g., [0]) is that they 'learn' how to reconstruct the images instead on extracting the 3d using optics.

Now the resolution is less a problem with plenoptic 2.0 cameras, in [0] they extract full HD images

[0] https://doi.org/10.1109/IC3D53758.2021.9687243

What is the basic principle behind this approach. A lens essentially allows you to map light rays (which have an origin and a direction vector) to position on the sensor (a pinhole does the same but lets through much less light).

Did Lytro have a sensor that could actually measure both the position and direction of an incident light ray ?

IIRC there was a micro-lens array between the objective lens and the sensor; each of the micro lenses added a separate, overlapping contribution on the sensor and with a bunch of signal processing magic you somehow back out the position/direction.

Reading the article, I don't at all see what the difference is between the Lytro camera and what's described there - although I'm not an optics expert.

One difference may be that, as I recall, Lytro was focused on (2D) photography, while this seems to be targeting 3D, which may be a much more compelling application.

I don't know - it seems pretty much the same - the whole point of the Lytro camera was that you had sufficient information in the recorded image to select your point of focus and depth in post-processing, which is essentially the same as having depth data.

The same information is used but there's a big difference between using it to make prettier photos versus reconstructing 3D scenes. The second application domain seems to have much more potential for revolutionary change than the first (to me at least).

The average person, or probably even the average VC, isn't going to read a pitch about being able to take photos without having to adjust lens focus and say "gee, with that technology you could build a 3D model as easily as taking a photo".

You more or less described it. If you put a microlens over, say, each 3x3 pixel sub-array, then each of those 3x3 squares now captures a locally incident light ray and its direction (with low resolution). You can now treat that as a single image pixel, but now containing angular information describing how that pixel changes with rotation in the field of view. Effectively you end up with an array of tiny cameras.

Applying the same principle in reverse, you can also create a 3D display by adding a microlens array on top of an image. If you use the same microlens array to capture the image as to display it (swapping out the sensor with a display, or a camera film with its developed photograph) then optical physics does all the necessary image processing for you.

The idea goes back at least as far as Gabriel Lippmann in the 1900s: https://en.wikipedia.org/wiki/Lenticular_printing#Gabriel_Li...

The article says this is an advance in the speed of processing light field images. Also the bit about seeing around an opaque object, which I'm not sure what that means but could be an interesting first?

> Also the bit about seeing around an opaque object, which I'm not sure what that means but could be an interesting first?

It's just a property of light field cameras.

Technically, other cameras can do it to some extent too. If there's some opaque object with another object behind it,

1. If you focus on the front object, you'll see the object behind it--blurred.

2. If you focus on the back object, you'll be able to see the back object behind the blur of the front object.

With a light field camera, you can essentially "cut away" the blurry parts and get them in focus. You're not able to see anything new, you're just able to see things in the original image more clearly.

Sure, but that's not new. I was wondering if they did something extra.

Applied Science has a good video[1] showing how you can look around objects. He is using a regular lens there, but with a lightfield camera you can emulate basically any lens you want in software as long as the captured lightfield is big enough.

[1] https://www.youtube.com/watch?v=iJ4yL6kaV1A

Biggest problem was motivation, you could instead capture the whole image and apply a digital blurring filter.

Reply on news.ycombinator.com