Smartphone Cameras Go Hyperspectral | Hacker News Reader

Smartphone Cameras Go Hyperspectral

spectrum.ieee.org

152 pointsvoxadam87 comments

This is really cool and very clever. But i want to raise one thing.

> designed a special color reference chart that can be printed on a card

My rudimentary understanding of physics makes me suspect this sentence is a simplification.

A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do.

So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands.

I suspect alot of effort went into finding, mixing and algoritmically combining the right inks.

I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp)

Insanely clever. (Whatever they did)

Not just that, but it would presumably be sensitive to light emission spectra too. As inks can only reflect wavelengths of light that hit them, if the emission spectra has spikes or gaps - think LED or florescent - the reflected spectra will be a function of the light source[1].

Perhaps there's some accounting for this, and I'm curious to learn what it is, because it's a phenomenally complex problem.

1. You might think the sun is a standard source, but it's usually modulated by the atmosphere[2].

2. Unless you are in space.

> Perhaps there's some accounting for this, and I'm curious to learn what it is

The slip itself is a calibration reference, so a clean photo of it could serve to compensate for the lamp and camera and calculate how accurate the readings is for different parts of the spectrum. (But good wide spectrum light would be ideal for high precision readout)

You're also still limited to visible light because of the camera uv and ir filter, for which the sun is a decent reference.

oh, yes of course! Thank you :)

I don't think the sun is even a perfect source when you're in space, doesn't it have gaps in its emission spectra from the gasses that make it up?

You are quite right, I had no idea.

"Between around 10,000 nm (far infrared) and around 100 nm (deep ultraviolet), the spectrum of the Sun's spectral irradiance agrees reasonably well (though not perfectly) with that of a blackbody radiator at about 5,700K. That is about the temperature of the Sun's photosphere. The deviation from a perfect blackbody spectrum is due to many factors, including the absorption of light by constituents of the solar atmosphere, and the fact that the photosphere is not uniform, but has some hotter and some cooler regions, so that what is seen from the Earth is a composite spectrum of blackbody radiators at a range of different temperatures. About 99% of the total electromagnetic radiation coming from the Sun is in the ultraviolet-visible-infrared region."

https://acd-ext.gsfc.nasa.gov/anonftp/acd/daac_ozone/Lecture...

Printing can use so-called spot colors.

I similarly thought that just because they said print does not mean it was printed on someone's ink jet. I'd hate to see how many different Pantone colors might be necessary.

If you only need one card per 10,000 photos, then the cost of the card starts to look cheap compared to a spectrometer and its bulk.

Ink is perfectly capable of being a phosphor, in which case it'll up or down convert wavelength X to wavelength Y.

My gut feeling is that finding enough very specific wavelength shifting inks would be harder. Perhaps its a mix though to get good readings in the faar edges between the rgb wavelengths.

I hope there is a research paper on this i can read.

I expect rolling shutters to be a far larger issue. Haven't read the article a second time, but non-uniform and inconsistent lighting is a huge challenge for this kind of work.

Wouldn't it be nice if they just told us so we didn't have to speculate? This is cool stuff and I'm glad I know about it but, as someone interested in this field of study, I'd love to try this out. But I guess I should stop being surprised when even a company like IEEE can't be bothered to write an article with any actual information. Just a bunch of simplified summarized crap.

At the bottom of the article is a link to the paper, which is open access.

I missed that! Thank you!

Based on supplementary material "We utilized a professional photographic inkjet printer (ImagePROGRAF PRO-1000, Canon), equipped with 11 ink cartridges and a chroma optimizer". Not your typical office printer but nowhere near as exotic as you might expect.

How special are the inks in the cartridge, though?

I have to assume it used inks available from Canon. Making your own inks is a research project on its own, so my guess is that would have been mentioned.

They are special in the sense that they are calibrated and QA'd for professional use

they are not four but twelve different (!) base colors. they are calibrated and are very light proof i.e. degrade much slower over time, "look the same" after years.

what's your background in understanding how printing works?

https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1...

We utilized a professional photographic inkjet printer (ImagePROGRAF PRO-1000, Canon), equipped with 11 ink cartridges and a chroma optimizer (PFI-1000 LUCIA PRO Ink, Canon), and used the manufacturer-recommended genuine paper (Photo Paper Premium Fine Art Smooth, Canon) for printing. To reproduce the desired reference colors for the spectral color chart, we also implemented a customized printing calibration process while maintaining the International Color Consortium (ICC) profile. The actual printed colors (output) showed notable distortions compared with the intended colors (input), which were particularly influenced by the type of paper (print sheet). For customized printing calibration, we mapped the exact relationship of the CIE xy chromaticity values between the digital color input and printed output values. After the printing process was completed, we measured the reflectance spectra of all reference colors from the printed spectral color chart (Fig. S1) using a spectrometer and a diffuse reflectance standard (equivalent to using CIE illuminant E). We confirmed that the CIE xy chromaticity values obtained from these measurements were in excellent agreement with the desired input values within the SWOP v2 gamut (Fig. 1(e)).

I would assume that with time, you could just print it on a generic $30 inkjet, and calibrate the card, sensor, and whitelte balance to a stored reference image.

It won't be as accurate, but it might be enough to offer some insights into whether liquid photographed in the article is in fact whisky, not urine (which to me seems to be a much more noble demonstration subject).

I was initially surprised too, but I think there is a some trick...

I'm not sure what happens on paper, but when you have ink disolved in water the abortion is not linearly proportional to the concentration, it's exponential. For example, consider a red ink and 5 magical selected frequencies and the absortions at 1% of concentration are: 99%, 99%, 50%, 10%, 1%

If you double the ink at 2% you get 99.99%, 99.99%, 75%, 19%, 1.99%

So increasing or decreasing the ink concentration may give you information of different frequencies, even with only one ink and only one sensor. In this case mostly about the 3rd and 4th. With more concentration you may kill all the light in the 3rd and measure the absortions ratio between the 4th and the 5th.

One problem I see here is how to order them, but I guess it's possible with a few sensors and a few inks. Each sensor sees all the frequencies, but with different weight. I'm not sure if it's possible to solve this, but perhaps you need some initial approximated model of the inks(???).

Now, when you put the inks on paper and have a unreliable light source and perhaps other technical problems, ...

In conclusion, I think it's possible to use different saturation and mixes of the inks to get different spectral distribution of the light that bounce on the card. Then use the three sensors to get three averages and try to use big linear algebra book to reconstruct what happens in between. But I should read the paper to be sure.

This doesn't really seem like "hyperspectral imaging". I think the idea is having a reference colour chart of known emission characteristics and photographing it through a transparent substance gives you an idea of how much that substance attenuates each wavelength.

It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?

In theory maybe you could build a version made of inks printed on a reflective mirror? And then you would hold the mirror so it reflected the object into the camera?

But that seems far more difficult. Precisely combining and applying combinations of inks to a mirrored surface sounds like a helluva manufacturing challenge.

I think ink and mirrors are kind of fundamentally incompatible.

Probably closer to what you're thinking about would be putting a bunch of tiny bandpass filters infront of a mirror, but in that case you can ditch the mirror entirely and just point the camera through the filter array.

A filter array right on-top of the sensor is how (the vast majority) of) CMOS cameras distinguish colour anyway.

On top of that it only works in the VIS range, thanks to the filters in front of the camera sensors - and most of the interesting information is in UV or IR. VIS only contains information about a few elements. (see Fraunhofer lines https://en.wikipedia.org/wiki/Fraunhofer_lines)

> The new patent-pending technique

> “Every photo carries hidden spectral information waiting to be uncovered. By extracting it, we can turn everyday photography into science.”

And with our patent, extract rent from anyone who wants to do it!

That's a bit of a bad faith take. You were welcome to go spend the years(?) this chaps dedicated to putting together the research required to build this. If it works, let him enjoy the fruits of labour.

Patents do more than let you enjoy the fruits of your labor - the market already allows for that. Patents use the force of law to bar anyone else who might have discovered the same thing from building upon it.

Imagine you are just a dude, you did all this work, and go to "market".

You are just a dude, therefore business grows slowly.

You gather enough attention that some corporation with a lot of bling just goes and copies your thing.

Your business fails.

On the other hand those same corporations can generate, file, and litigate more patents than just a dude could ever hope to.

It's 2007. Just-a-dude has a great idea, he notices customers to his website often buy just one item, so he'll let them do that with one simple click. What's this, he's just received a cease and desist? Sorry bro, Amazon patented that 10 years ago.

Once can just as trivially construct an argument demonstrating the issue with patents but the problem with this style of argument is that patents are not a simple thing. They have global far reaching effects. The government distributing a monopoly on information is a serious interference with the market, and due to patent harmonization efforts across the world, one person filing a patent in New Jersey affects even people in Kenya and Turkey and Thailand. The arguments for patents are often, as I see it, based on a deeply flawed understanding of the motivations of innovators and the affects of open information on innovation. For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.

>For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.

Can you clarify this? Just curious about what you mean.

I mean it sure sounds like the old “science can’t explain how bumblebees fly, therefore science is incomplete or outright wrong” argument. Which is of course just false.

I replied so feel free to read that.

I often speak to people who say that no one would innovate without government-secured monopoly on information, and this is clearly false as open source works without this monopoly. As such it seems to me that people who make that claim don’t understand how innovation works.

Often I say that if we phased out patents and other IP restrictions, then investments would not stop but they would change from less frequent large investments to more frequent small investments. As long as designs can stay secret until release, there will always be first mover advantage and brand recognition. But you might get smaller investments to build out manufacturing for the next year, rather than bigger longer term investments. The flip side is that stagnant innovators who got lucky once will be subsumed by more agile competitors who can better deliver those innovations to market.

Thus investment and innovation wouldn’t stop - markets still ensure certain advantages for innovators - but the nature of investment and innovation wouldn’t shift towards more incremental moves and more diverse actors. A major upside to this is that those best suited to scale an existing nascent technology would be free to compete at doing so.

It should be noted that even die hard capitalists are against IP restrictions [1] as they are a massive government investment in the market. So proponents of IP restrictions must reconcile their arguments in favor of this government intervention with their potential interest in free markets.

[1] https://youtu.be/GZgLJkj6m0A

Yeah, I do agree with that. Same with the rationale for copyright.

I mean we’re basically getting the same result. Tons of businesses, not to mention patent trolls, constantly harass individuals and small businesses trying to get their foot in the door or just run a small, sustainable business. Hell forget my business failing, it’s possible I’ll never even get to try my idea out!

you have it entirely backwards; patents dont protect just-a-dude, they protect the corporation.

how?

just-a-dude doesn't have a team of patent attorneys sitting in his back office waiting for work.

Sure, if he'd come up with this primarly using his own resources and time, but he discovered this whilst being paid to conduct research at a public university, a form of institution which is explicitly intended to disseminate knowledge. Society should enjoy the fruits of its investment.

Would the university have been willing to invest in the exotic printer and the labor to do the work without the potential upside of a patent?

It’s very easy to declare that someone else “should” do a bunch of work and spend a bunch of money for the altruistic benefit of society.

There is not much labor here. Anyone should be able to engineer a model by using deep learning over pictures that map raw images to their hyperspectral variants in various settings, including in adversarial settings that are intended to confuse. All you would need is a sufficiently large and diverse dataset.

I hate the new rhetorical use of the word “rent”

"Rent-seeking" is square 3B on my HN bingo card. See also:

  Regulatory capture
  Late-stage capitalism
  Walled garden
  An/The unreasonable...
  So...
  Period. Full stop
  I Mean…
  Streisand effect
  orthogonal
  trivial
  non-trivial
  {$person}'s Law | Axiom | Razor | Paradox
  Objectively
  Ship of Theseus
  Gatekeeping

It’s like the opposite of traditional tech buzzword bingo where you raise VC for blockchain crypto AI agents.

Instead, anything you don’t like is a rent-seeking late stage capitalism narrative.

I seriously doubt using the word rent in reference to patent license royalties is new

I notice the article doesn't say anything about accuracy. This is not my area, but I think the _other_ hacky way to try to do spectroscopy with a phone is with a diffraction grating (and maybe a box with a slit in it). Diffraction gratings are cheap, probably not so different from a specially-printed reference card. If you have a choice, which is better?

diffraction grating wouldnt give you a controlled lighting environment (illuminant). they seem to handle that issue here by using a known spectral reference chart which might let them handle any normal lighting environment.

I would think in the same environment you would take images immediately before and after adding the sample.

The actual article does say quite a bit about accuracy.

I do not understand how this could possibly work. From a camera with RGB filters we get essentially three different integrals over the spectrum per pixel, how would we recover the spectrum from that? Even assuming you can account for the spectrum of the light sources and the color filters in the camera - which should be doable with a color chart with known spectra - I do not see how you could go from three data points to a full spectrum without making assumptions about the possible object spectra.

EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.

I don't understand how from 3 independent values per pixel (RGB) they claim to derive 200+ independent values per pixel. Unless they are assuming a smooth "image" (all pixels the same RGB), perturbed only by the color card? Not exactly a camera then

They're not claiming to get that many values per pixel, they're getting that many values overall for the medium through which light passes between the card and the phone. The idea light comes from a source (e.g. sun), bounces off the various colors of the card and thus produces hundreds of different spectra, those all pass through a medium, and land on the phone camera. So you're getting one measurement consisting of hundreds of RGB values that each represent intensity of different spectra, and you combine it all together to get a single spectrogram.

Did you read the actual article. They go into the method they use at length.

More specific info on the reference card is available in the paper's supplemental information. https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1.... Basically, they used special paper with a pro-Canon inkjet, along with a special ICC color profile.

Someone needs to build a phone that is leaning towards a tricorder; I'd buy that for myself and my kids. My Pixel 10 has a temp sensor on it, which is cool, but I've had minimal use so far.

I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).

Take a look at the phyphox app - https://phyphox.org/

Wow! That app is amazing thanks for sharing

This is brilliant.

Also Arduino Science Journal: https://www.arduino.cc/education/science-journal

Caterpillar has a smartphone with a thermal camera. The price isn't far off the the price of the most expensive smartphones

https://cat.smartwalkie.com/store/products/cats62pro

how does Cat's self repair policy compare to John Deere's? Then again, it's not far off from Apple's

There are also cheap ($200-$400 range) usb-c thermal cameras specifically for phone use (they're cheap because they're just a sensor, the app on the phone is the "screen" and controls.) Great for narrowing down overheating hardware, and you can keep one in a pocket.

Thermal is interesting, but hyperspectral is much wider than just thermal. The two are not the same.

I want both :)

I will say I have a FLIR C5 dedicated thermal camera, which I got used off Ebay for ~$300, and it's pretty useful to have around as a DIYer.

This could improve chromatic adaptation of captured images. In other words, better results when changing the white point.

Funny enough, that’s what photographers are doing when they shoot a color checker chart (e.g., Munsell, Macbeth, X-Rite).

White balance is hard, in part, because the sensitivity bands of our vision and the camera sensors do not align. Take a look at fluorescent (or, better still, sodium vapor) light spectra for clarity on why this is a massive pain.

Every picture contains noise.

Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one).

It makes one wonder how much of the noise is actually misinterpreted signal.

I was hoping that someone came out with a camera that not only had not only sensors for visible light, but for infrared and UV. It's just another color to add to the sensors; I think we have enough megapixels, seems like going for other bands is reasonable.

I have a OnePlus 8 Pro with an IR camera. It's pretty nifty - nature photography looks cool, seeing through stovetops is neat (and seeing when they heat up), and VR things are also often playing around with IR (plastic transparent to IR, IR LEDs, etc).

I ended up having to flash Lineage, as there was some outrage that in a highly limited set of circumstances, thin see-through T-shirts became slightly more see-through and OnePlus disabled that camera in their later firmware updates.

You have to love when amazing innovations disappear just in case the lowest-quality rung of our society might misuse something... I'm pretty sick of being ruled based on the lowest common denominator.

Some phones had near-IR camera (Pixel 4, Samsung S10) accessible via API. No "killer app" was found since then, 5+ years

iPhones also have a near-IR front camera, but that one is fully slaved to the FaceID system. Don't think anything in userland can access raw data from it.

There are lots of 3D scanning apps using "Face ID", like Heges: https://hege.sh/

Those rely on the depth maps, which can be accessed from userspace. But the depth maps are derived from IR camera footage, which is not accessible.

Ironically, older iPhones have better depth resolving capability overall. Apple sacrificed depth sensing performance in favor of smaller unit size in the newer ones.

I know many full size cameras have filters to specifically remove IR and UV from the images. Is this true for smartphones as well?

Yes.

CMOS image sensors are naturally sensitive to near IR. Early feature phones had no IR filters on their cameras - you could see an IR remote light up through them. But as people became more and more obsessed with smartphone camera quality, smartphones started to ship with those filters too. You get more "lifelike" colors that way.

Although in some multi-camera smartphones, one of the secondary cameras may lack an IR filter.

One of mine definitely lacks such a filter because I was able to catch not only the remote, but also an electric stovetop while it was still heating up and its glow was barely visible with the naked eye.

Pretty much all digital cameras and phones see a bright enough source like an IR remote LED. Filters don't remove the radiation, they just turn it down.

Back when I was in oil and gas, we were thinking of using modified mirror less cameras without and IR filter for vegetation density calculations. There were a few vendors that sold the UAVs and modified cameras.

Nowadays, there is a more mature ecosystem, with specialized drone mapping cameras tailored for the purpose.

For our use case, the micasense rededge would have been perfect.

Same! I want to be able to capture more of the spectrum already!

Hyperspectral imaging see things the human eye can't: Sure

Custom processors to accelerate generation of AI text: Go ahead

Slightly thicker to fit a bigger battery in: How dare you

This is a combination of things that almost breaks my brain, but it works, and it's brilliant!

First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time.

Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence.

The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable.

Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure.

[1] https://www.youtube.com/watch?v=_ezhdhHNku0

[2] https://en.wikipedia.org/wiki/Coded_aperture

[3] https://en.wikipedia.org/wiki/Fourier_transform

[4] https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllc...

[5] https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZ...

[6] https://en.wikipedia.org/wiki/Bayer_filter

Patent-pending... again someone trying to rent-seek a high-school physics fair idea. Measuring light absorption with a camera is almost as old as the camera.

Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique.

What "investment" is patent law protecting here?

What you're referring to is color calibration. This is spectroscopy. This is likley more of a chemistry paper than a engineering paper because the ink in the reference chart is doing some heavy lifting.

Reply on news.ycombinator.com