Simultaneous Imaging and Energy Harvesting in CMOS Image Sensor Pixels | Hacker News Reader

Simultaneous Imaging and Energy Harvesting in CMOS Image Sensor Pixels

techcrunch.com

132 pointstouristtam36 comments

This seems like it could be combined with battery-free "no-power" WiFi tags [1] to provide functionally indefinite surveillance at a minimal price tag.

While 1 kb/s bandwidth and a range of only 2 m limit the current applications of such a combination, the potential once both technologies become more mature is staggering (and rather terrifying).

[1] https://www.sciencedaily.com/releases/2014/08/140804134215.h...

There has been some recent work[1] towards backscatter video (actually from the same lab you linked to). The basic idea is the same as the "great seal bug"[2] except for photodiodes. I say "towards" because much of the work they present is in the form of simulation.

[1] https://homes.cs.washington.edu/~gshyam/Papers/videobackscat... [2] https://en.wikipedia.org/wiki/The_Thing_(listening_device)

Doubtful. Several companies have already come and gone with solar-powered security cameras that never really worked as desired. If we can't power a simple low-res (720p) wifi camera reliably with a large (as compared to an image sensor) dedicated solar panel today, this technology is unlikely to change that in the next decade.

You still need to process or transmit the images, which might be costlier than recording the images.

For transmitting data on an energy harvesting budget there is stuff like LoRa, but Image is out of the question here because of the size.

Much more likely is that such devices will be used for tracking, in situations where GPS is using to much power. Especially, as the calculations for that are done at the receivers instead of the transmitter.

"It turns out that photosensitive diodes aren’t totally opaque — in fact, quite a lot of light passes right through them. So putting the solar cell under the image sensor doesn’t actually deprive it of light."

Does that also mean we could put multiple layers of photosensitive diodes under one another to perhaps get higher resolution images? or better images in the dark?

Such a sensor (used to) exist, Foveon X3. It is very interesting as it also uses the differential absorption of different wavelengths of light at different depths in silicon to distinguish colours: https://en.m.wikipedia.org/wiki/Foveon_X3_sensor It had various advantages, and some disadvantages related to distinguishing colours, but ultimately wasn’t very successful in the market despite getting shipped in some DSLRs, possibly due to the marketing challenges (Bayer array sensor manufacturers count each R/G/B sub pixel as part of their “megapixel” count, so the Foveon sensor appeared low resolution by that metric).

My Sigma SD1M still produces mind-blowing images I can't do with high-end Nikons and Canons. Foveon can truly make magical images, though there were so many issues with DSLRs it shipped on, e.g. they were almost always awfully slow and had very bad (but film-like) low-light performance. Also, original research suggested that to get realistic colors, up to 7 layers are needed. Bayers OTOH just interpolate and cook colors, but with the increase of resolution might at some point perform similarly, when lens limit is hit. Foveon was also always manufactured by a far worse process than e.g. Sony top-end sensors, and its quantum efficiency was orders of magnitude worse.

Bayers don't cook colors when they interpolate, they cook edges.

I don't follow, Sigma hasn't stopped using Foveon sensors, they still use the Foveon Quattro stuff afaik. Now they use a 1:1:4 ratio for R/G/B layer resolution, where the blue layer also records luminance.

They decided they wanted to confuse people more when they tried to compare it to a Bayer sensor.

Actually I didn't realise they still made them... Nice to know. I remembered the Foveon sensors from when they were first introduced, but didn't see any reference to them still being made when I did a quick search to refresh my memory, so they may have a bit of a marketing issue ;)

Higher resolution - no. Resolution is limited due to diffraction, so adding more pixels doesn't necessarily add more details.

Better images in the dark - possibly.

More important than the diffraction limit is probably the fact you put an image sensor in front of your image sensor, and it will distort the wavefront for the sensor behind it.

Plastic bags are also quite transparent, but you never get a better picture by putting them in front of your camera.

Isn't it possible to computationally counter the effects of diffraction and distortion?

Generally speaking yes! That's what companies do that create masks for lithography so that the structures the manufacturers can create are smaller than the wavelength they use in the process. (See https://en.m.wikipedia.org/wiki/Computational_lithography). The article states that generally this is rather computationally expensive and I would wager that an analogous concept would be too. Perhaps a Neural Network could be trained to infer corrections and maybe that would be feasible then but maybe someone else with more experience in the field can elaborate on that.

Neural networks suffer from provability issues, you don't want to make a multi million dollar mask set only to find out your ConvNets were hallucinating. There are more standard computational lithography techniques such as OPC, RET and inverse lithography.

With that being said, there's been some work done:

https://ieeexplore.ieee.org/document/5726499/

https://www.spiedigitallibrary.org/conference-proceedings-of...

http://iopscience.iop.org/article/10.1088/2040-8978/12/4/045...

https://www.google.com/url?sa=t&source=web&rct=j&url=http://...

Non-mobline link:

https://en.wikipedia.org/wiki/Computational_lithography

Especially when you have a predistortion image from the frontmost sensor, and I would imagine that the diffraction would remain constant between photos - manufacture differences would presumably mean each sensor would need to be calibrated, but with the multiple sensors, it seems quite possible that could be done by the consumer.

They say the answer is "yes". It's even nominally offered as a feature in, e.g. Canon's RAW processing tool (called "DLO")

Unfortunately, in practice it seems to behave more like smart sharpening than a discrete reverse convolution math op.

Already on the market. Interview with Foveon cofounder Richard Lyon : https://www.youtube.com/watch?v=pIZp0qPZySw

Fun fact - Carver Mead was one of cofounders, of the Conway-Mead fame (founder of modern VLSI revolution). Another was Faggin, father of Intel 4004. Whole company was loaded by hardcore semiconductor process design engineers.

Although I couldn't stand The Circle (book or film), it did raise questions about what happens to society when we can place extremely cheap cameras everywhere invisibly that require no battery, and hence can operate almost indefinitely.

In The Circle they are owned by a pseudo Goog-book company that publishes (and records) every video feed (and its history) publicly.

Another book to check out is Vernor Vinge's "A Deepness on the Sky," which depicts our smart dust future in a way I still think about all the time, decades after reading the book. His "Qeng Ho Localizers"

- are invisibly small (1mm square and thin)

- can be released like a gas into the atmosphere

- are powered by low-power, low range microwave bursts

- mesh network with each other

- have omnidirectional video, other sensors

In the book, historians survey the wrecks of myriad advanced societies in our local galaxy area. The find that smart dust, as final-form state surveillance, is a kind of an end game of technological development. That is, it inevitably emerges, and inevitably destroys society.

On the brighter side, if you can power/harness the localizers around you, you can expand your sensorium to sense everything they do, which sounds cool.

You might enjoy "The light of other days" [1] then.

It envisions the world where wormhole cameras can view any event in the past, thus eliminating all secrecy and privacy overnight.

[1]: https://en.wikipedia.org/wiki/The_Light_of_Other_Days

I started to wonder about this, but then I realized that what we're really witnessing right now is the birth of the primordial Grey Goo. I mean, omnipresent super-observation states may be one angle of Our Future Dystopia, but I think energy-harvesting super computers that can reproduce is right around the corner on this one ..

Reproduction is one thing I'm really not concerned about actually. Is it even possible to build any kind of silicon-based electronics outside of a fab?

We've already entered into a period of symbiosis, where software controls human bodies. It seems unlikely that machines can ever do better than a human hand (but can do better almost everywhere else!)

That software is still created by humans, subject to human forces though. You can characterise many things as 'training' or 'evolving' humans to serve their purposes, e.g. gut bacteria, poppies, cows, dogs, toxoplasma gondii, hammers etc. The characterisation doesn't tell you much though.

There was a science fiction story about a society where privacy was completely impossible because air everywhere was permeated by nanoscale cameras that recorded everything everywhere from all perspectives at all times. Looks like we are approaching that future.

Those would be useful for vascular and other medical imaging uses. Search and rescue bots as well.

If you scroll down too far the website hijacks your back button

But a camera still needs some room (behind a point hole, and potentially some lens). At some point this will have higher impact than sensor size, no?

You know what a cool application of this would be?

Cameras in your bloodstream that create a distributed 3D image of your entire circulation at once.

What would the light source be?

A 50w LED that follows along externally?

Bring it on

Yeah !

Reply on news.ycombinator.com