Hyperspectral analysis with just an app
fraunhofer.de
fraunhofer.de
The headline "hyperspectral analysis with just an app" is unfortunately kind of bullshit. (And Fraunhofer seems to have realized this, as the current headline is just "App reveals constituents".) As highd points out, this will give you nine spectral dimensions instead of the usual three, or eight rather than two after you drop luminance, which will almost certainly be the first principal component. From eight spectral dimensions you can maybe get back to eight wavelength bands.
(And you do get nine spectral dimensions rather than the usual three, because the spectra of the light emitted from the display are not going to exactly match the response spectra of the pixel colors in the camera. But they'll be close, so some of those dimensions will have very little energy, so they'll be very noisy, and JPEG compression is likely to be fatal. And of course they vary from one device to another.)
An eight-band image is not a fucking hyperspectral image. It's multispectral, like Landsat. But that's still enough to do quite a bit of material discrimination. Probably not even close to enough to detect pesticide residue, though.
The only way you could get more information this way would be through nonlinear effects, where illuminating the object with twice the light gives you a scattered result significantly different from twice the original scattered result. Those don't contribute significantly to the spectra of ordinary materials except under extremely intense lighting conditions, although I've seen two-photon effects do interesting things with glow-in-the-dark paint and a red laser pointer, and of course there are devices (like green laser pointers!) that take advantage of the Kerr effect — by using exotic materials.
[1]https://www.infosecurity-magazine.com/blogs/ultrasonic-cross...
[2] http://kenstechtips.com/index.php/how-to-see-the-invisible-i...
An LCD has a totally different spectrum (broad) than an LED display (single wavelength). Cameras on the other hand always have a broad spectrum, but it varies depending on the color filters, the response curve of the pixels as well as the post processing (e.g. the demosaicing algorithm). Regarding response curve of the display: that's also something they have to calibrate for and it's not only a hardware parameter because it can be modified in software, for example by the iOS Night Shift feature.
Making this work with a single piece of hardware is hard enough and I wonder if they'll target the android market at all. My guess is they're going to release it only for iPhones because it's so homogeneous.
Also, the press release photo made me giggle, the front camera isn't even facing the object they are scanning :D
Edit: Saying LEDs have a single wavelength is probably not correct, but they have a very narrow band depending on the technology used.
In this case, you need to simultaneously estimate the LED wavelengths as well as the response curve of the camera.
Not saying it'd necessarily be worthwhile to do, but if that actually worked, all you'd need to do is send a light source + a color swatch booklet through the mail to a potential user if the device isn't in the database yet.
If that's true, and those constraints aren't terribly well understood, it would make for a good machine learning problem. Fwiw.
The only thing we can say for sure is that the quality of the result varies from device to device - if it works at all.
I suppose if the three screen LED wavelengths were significantly different from the three camera filter wavelengths then you could:
Illuminate with screen Red to get Rscreen.
Illuminate with screen Green to get Gscreen.
Illuminate with screen Blue to get Bscreen.
Use ambient full-spectrum light to get Rfilter, Gfilter, and Bfilter.
Then you'd have 6 points on the spectrum.
I'm not sure how their inverse algorithm could work, but I have a feeling it should be possible to get more than three points of the spectrum by displaying multiple light patterns.
Regarding full-spectrum ambient light: they can't use it at all because they have to subtract it from the images. You can only recover spectral information from the light you control. At least that's what I'm thinking right now.
There's also no way you're measuring pesticide residue with that - I doubt that would even be possible with a high-end visible hyperspectral camera. Maybe with a raman spectrometer.
I've designed a couple versions of cell phone camera-based spectrometers and spectral imagers, so I'm relatively familiar with the design principles.
Are you sure that it is impossible to detect it, even if the app is 'calibrated' to a certain object, e.g. an apple? I think if you limit the search space it could be possible!?
Most chemicals have characteristic spectra in the infrared, so you'd need sensors going to much larger wavelengths to have significant difference - even then it would be hard to detect against the variation in signal from fruit.
Spectroscopy is predominantly done with UV (200-280 nm most common) and IR, which are regions where photonic interaction is dominated by electronic and vibration/rotational transitions, respectively.
Visible light absorption is typically caused by highly conjugated bonds and metal-coordination complexes. In terms of day-to-day, this is almost exclusively dyes (synthetic and natural). Dyes also tend to be really potent absorbers - you only need minuscule amounts of them to create very vivid colors. So a purely visible-light-based app would at best be able to give you a handle of what sort of dyes are in something. It won't tell you if it has pesticides (let alone traces!) or HFCS or nutrients or what-have-you.
tl;dr - no, it's not remotely possible to even detect pesticides with visible light.
I also build a swarm of throwies that used a single LED to synchronize their blinking. That project was much cooler, and I wish I had better documentation...
I suppose I should recreate it, on an even larger scale. I just need to figure out some better way to power it...
I'm guessing you didn't try to calibrate the response curves of the LEDs (in both directions)? I'm not sure if the desired effect is even feasible (assuming the receiving bandwidth is as narrow as the as the transmitting one), but it should give better results? And if it's not possible at all, you'll surely notice it during calibration?
I'm anticipating someone snapping an image and an app saying, "That's either a fresh organic grape, a tractor tire or a leg of lamb."
Will you have to tell it: "This is a Kale leaf" and let it evaluate the signal levels relative to other Kale leaves?
It is certainly not going to be precise, but it's quite an achievement if they can may it work within the limits of the device.
Spectrometers generally produce a single spectrum from a light source.
A hyperspectral imager produces a spectrum for each pixel in the image. See https://en.wikipedia.org/wiki/Hyperspectral_imaging
Like with a grating I can see how you can obtain a spectral pattern for many wavelengths, but with a bayer array, surely you can only distinguish R, G, B.
Would I be right though in thinking, if there's no stray light and only the light from the phone screen shining on the apple, you can cycle through many wavelengths of light, enabling you to measure the reflected light to determine the spectra.
One thing I was wondering so pesticides can presumably also be detected using visible light spectroscopy then?
They inversed the principle: Rather than controlling the bandpass of the sensor (which a grating does passively, over the spatial domain) they are controlling the spectrum of the light source. You can produce a varying light spectrum by setting R/G/B on the display, and record the light reflected by the object with your R/G/B camera pixels, which also have a certain frequency response.
I have a feeling you can use this to recreate a coarse approximation of the full spectrum and then use it to infer some of the object parameters, like pesticides on fruit.