A DIY near-IR spectrometer | Hacker News Reader

A DIY near-IR spectrometer

caoyuan.scripts.mit.edu

365 pointsradlad73 comments

This write-up and project is from the same guy that discovered superconductivity in bilayer twisted graphene. Pretty impressive.

I don't know how uncommon that is, but more than 14k citations at age 25, most of which as first author sounds pretty darn impressive.

Not knocking his efforts, but I guess its on a par to writing some code or making something which is used by many people.

I once went through quite an experience. Used to work with a guy that was awarded a big prize in his field. I think one day he got tired of being treated as "some kind of genius", as he said. He started to discuss a topic one day, listening to my answers and questions back and forth. At the end the discussion hit a point where we could not go further. He then told me: "if you then submitted this as a proposal, were lucky enough to have it granted and just answered the last question you asked me with a simple experiment, you would receive the same prize as I did".

That guy was great.

You're probably being intentionally vague here to avoid "naming names", but I'd love to hear the actual details of this story: the topic under discussion, the interrogation that lead up to the "point where [you] could go no further", what the last question was, and what a simple experiment to answer it would look like.

And a genius lol. Don’t tell him I said that

Probably underestimates it. Not my field, I don't know of this guy but a few hundred citations is you write a library that everyone uses and similar academic recognition and academic platinum for tenure on the background of other publications. 300 000 is the equivalent of writing the Linux kernel or sqlite.

https://www.nature.com/news/the-top-100-papers-1.16224

It really isn't -- unless you're talking something like bzip2 or some extremely nontrivial thing.

i've written code used by at least a billion people over fifteen years. i'm nobody and it was nothing. so definitely not on par

That really is knocking his efforts. How is that constructive?

Right - that’s impressive

Aside from pure curiosity and or research purposes... Is there any everyday/practical apps/uses for something like this, for the avg joe? Building something for 500 that would normally be 10k, already has me intruiged.

I brought this up in a comment in another thread. I found this page because I wanted to see whether it was feasible to monitor nutrient concentration in an aeroponic solution. Typically the solution circulates between the tank and the root box, so the concentration in nutriments will decrease gradually. Near-IR isn't enough to figure out these details I believe (I'm not a scientist), but it is enough to get a fingerprint of the nutrition "equilibrium". These solutions are probably the most technical aspect of aeroponics, so I wanted to use commercial solutions bundled as three bottles you're supposed to mix in various proportions. I think that if you explore the concentrations space using such a spectrometer you may be able to train a model to interpolate the concentrations from spectrographs measured in "production".

Anyway I think this is best to address the shortcomings of circulating the solution, for instance use a tiny tank, circulate for a few hours/days, refill, dosing as you go. A spectrometer would be still be cool in order to get real-time concentration diffenrential for each solution component. Identifying individual chemical species could be accomplished I think using far-IR spectroscopy with a model trained on data measured using more expansive spectrometry techniques.

I haven't grown anything yet, I want to build a sap flow sensor first.

https://edaphic.com.au/products/sap-flow-sensors/

https://dynamax.com/products/transpiration-sap-flow/dynagage...

this sounds like a relatively cheap project.

* any microcontroller with multiple ADC channels, for example ESP32 if you want it to be wireless as well...

* a heat generator (can be as simple as a transistor)

* a few thermistors, which you will have to calibrate (pretty easy if you have access to a thermometer, an electric kettle and ice, basically dunk the thermistor in a glass, put the thermometer in the glass, have the microcontroller output measurement number, and measurement value over serial port, now heat water to boiling point, make a table with a first column pre-entered with the thermometer gradation, then pour it in the glass, each time it passes a gradation you look at the scrolling list of values and note it in the second column of your table. Now you have a monotonically decreasing list of measurements, some of them marked as a thermometer gradation passing. Make a preadsheet and use the value of the fixed resistor in the voltage divider [the other resistor was your thermistor] together with the standard NTC or PTC formulas, then fit the free variables of the formula to the data... An acquaintance of me was starting to brew beer had all the components, but didn't know how to calibrate it, so I helped him out. Doesn't take long if you set your mind to it and properly prepare.)

Perhaps the difficulty is in the mechanical fabrication such that you thermally isolate from the phloem (which you puncture) and selectively measure temperature in the xylem?

Perhaps standard thermistors are too large. so an alternative would be a tiny diode, whose IV curve is temperature dependent. Added benefit is lower thermal mass, although I was surprised at the response time of the thermistor once calibrated (and filtered to result in step transitions when putting in and out of hot water and icy water).

EDIT: just making sure, what I'm trying to say is, don't be intimidated, and break up the task in smaller tasks, go ask for some help on ##electronics or so when you have sketched an initial plan, and listen to their feedback.

I used to work doing software for fruit sorting machines and we had a NIR Spectrometer to work out the sweetness of fruit, so when I'd go to actual sorting sites, especially in California which have these machines on a massive scale, I'd use it to sort out the biggest sweetest pieces of fruit.

That sounds super interesting. How does it work?

I read a lot of articles of this nature, not necessarily in this specific subject area, and may be guilty of writing some myself. I rarely build anything exactly to its original plans, but I take away bits and pieces, theory, and techniques, and re-combine them for my own purposes.

The value of the whole thing hanging together as a product is that you know the person has worked out the bugs of the individual parts, to the point where the whole thing is testable and reproducible.

This is actually true of the scientific literature as well. People rarely reproduce entire research projects, but often borrow bits and pieces and adapt them for their own use.

I don't know enough to know if this one would work for this kind of usage but cheaper and easily accessible spectrometers would be amazing for harm reduction and drug testing usages.

You could run Quality Control on your amazon chemical purchases if you were willing to put in time to find methods using spectrophotometery in this wavelength range.

Wow this is the coolest thing. I was looking at trying to build a DIY raman spectroscope and you can make one of those for under 100$ [1], but near-IR spectroscopy opens up more possibilities for sure. From what I understand (I'm not a scientist but I love the idea of having access to these tools) raman spectrscopy is limited to asymmetric crystalline molecules, so you can't really get a good reflection off of say salt to get a reading. (Edit - Looks like Near-IR is complimentary - "Raman active vibrations aren’t visible in the infrared for molecules with a symmetrical stretch. Similarly, infrared active vibrations aren’t visible in the Raman spectra for molecules with asymmetric stretch. This is known as the Principle of Mutual Exclusion and is what makes NIR and Raman Spectroscopy complementary techniques." [2] ) What we're really hoping to find out is if we can detect stuff like pthalates in plastics and heavy metals such as cadmium in metal products, I understand we'll probably just want to send things off to a lab to get GC/MS done for those things after talking to some people who have more experience than I do but I still want to see if I can build a bunch of these raman or near-IR spectroscopes for pharmacies in the third world. Counterfeit drugs are a real problem out there and they don't need to be.

[1] https://www.hackteria.org/wiki/images/a/a0/MobPhone_RamanSpe...

[2] https://www.labmate-online.com/news/mass-spectrometry-and-sp...

Thanks for the link! I’ve been trying to figure out how to buy/make a Raman spectrometer for cheap (currently in a third world country too!). Have you built one yet? I’m having trouble finding the lenses needed (mostly because of my lack of knowledge). Any chance you know what to use?

Laser ($40): https://a.co/d/0wjNGBz

Diffraction grating ($12): https://a.co/d/6bpO8xm

Laser focusing lens: Not found

Fluorescence collection lens: Not found

Focusing lens: Not found

Collimating lenses: Not found

I haven't found the lenses yet although supposedly https://www.thorlabs.com/ has them and someone else recommended these guys to me for finding more high quality diffraction gratings than the 'rave goggles' quality you can find on amazon https://www.edmundoptics.com/

>>While a silicon-based camera is as cheap as dirt these days, an one-dimensional InGaAs pixel array already costs upper few thousand dollars. Any full-blown IR spectrometer system goes way over $10k, with their fancy thermoelectric cooling and precision gratings (we actually have one in our lab). The reason why they are so expensive is that the target user group are scientific researchers, not consumers. <<

the expense is a product of intended use. performance guarantee, is responsibility for design failure. scientific instrumentation must be very consistent for the purpose.

Yea ok but…

Certain things just need an answer quickly. Like, some science that you do in order to do other science. And you can’t get there without a $10,000 device, you can’t even get started with some other experiment.

There isn’t anyone making hobbiest optical sensors. I looked for a project I needed a laser for. My options were, don’t do it, or pay $4000 for a late 80s sensor I needed off eBay then try and figure out how the thing worked with no software and a proprietary cable. Sound like fun? Not to me. And I didn’t have $10k for the USB version from the early 2000s, again with no software. And obviously far from thr $50k for a UBS3 new model.

Optical equipment is stupid expensive and sometimes you just need “basic tools” for advanced things.

Well sure, but with time prices drop, and with lower prices, you get even unscientific uses.

Flir units were for army use, now you can buy cheap chinese ones (without US export restrictions) for a few $100 in a form of a dongle that you plug into your phone and see what part on your PCB is overheating, or where your house insulation is bad.

Keeping liquids at precise temperature was a thing used in labs, and now you can buy a sous-vide machine for $50.

Even medical stuff, such as blood oxygen meters are at single dollar/euro prices now.

So yeah... it's a circle... if prices go down, more people will find more uses for it. If more people find more uses, more companies will produce them and prices will go down more.

One aspect in the price of scientific tools is alignment with know physical units and breadth+flatness of measurement. If your goal is to control a dynamic system within a narrow band, you may cut on price to fit for your specific needs.

The key implementation to save the cost is this:

> Next comes the detector. How are we gonna detect light from a range of angles with only a single 'pixel' of photodiode? We mount the photodiode on a motor and scan it across! To do this, I purchased a stepper-driven linear stage from Amazon for $50. This little stage is quite well-built and allows a resolution-per-step of 5 um, more than enough for our resolution.

So instead of using an expensive array of InGaAs sensors, he uses a single sensor ("pixel") and use a motor to physically scan across space.

For my previous work, I used industrial and scientific IR spectrometers costing above 100k USD. While the sensitivity of the physical machines is important (especially for basic scientific research), it's often the software that makes or breaks the usage case. Modern spectrometer software contains advanced signal processing and machine learning.

Maybe we should have more fundamental science collectives where amateurs can also contribute, in addition to academic and state funded institutions. Sort of like amateur astronomy clubs [Amateur astronomers still contribute widely to their field(s)], in larger scale and contributing in much more significant ways.

Yes to more amateur scientist articles on HN!

As you mentioned, astronomy is a field where contributions by amateurs / citizen scientists are extremely valuable. A few organizations that exemplify this are: (1) the American Association of Variable Star Astronomers (disclaimer, I'm their Executive Director) which collects photometric (brightness) data as well as spectroscopic data on variable stars and kindred objects including exoplanets (2) the Society for Astronomical Sciences which is more broad than AAVSO, but with a moderate focus on instrumentation (3) the Center for Backyard Astrophsyics (hyper specialized on one type of variable star, a good collaborator of the AAVSO) (4) the International Occultation and Timing Association that observe asteroids occulting (blocking) stars to infer their shapes

Most countries have organizations similar to these too.

Edit: There is a group within the SAS working on an automated optical spectrograph for astronomy called the FlexSpec 1 (https://flexspec1.readthedocs.io/en/latest/). It is about $500 in parts. Similar devices sell for about $3,000.

How plausible would it be to contribute if you live in fairly bright suburbs?

Suburbs aren't as bad as you might imagine. I observe from my patio under Bortle 6 skies (you can looks yours up at https://flexspec1.readthedocs.io/en/latest/) with a 5" telescope. We use to have a robotic telescope located in Cambridge, MA and it produced useful science.

The author's clearly not an amateur. He's a condensed matter postdoc.

I think there's an argument to say that he was doing it as part of his work, but I'll make this point anyway: if he was doing it for fun and to entertain people in a way unrelated to his actual job, wouldn't it still be "amateur science"? The same way that if a premier league footballer or an MLB hall of famer went with their family to play a game in the park, that's amateur football/baseball despite being done by a professional.

And while one could argue that the words used were "amateur scientist" rather than "amateur science", it could also be argued the opposite direction by suggesting that tag might not actually apply to the author but instead to the type of content since it's to be read and perhaps replicated by amateur scientists more than professional ones.

Or we could just celebrate the article regardless of the accuracy of someone's throwaway remark's categorisation of it :p

> more fundamental science collectives where amateurs can also contribute

Look anywhere there is field work out in nature, there's amateurs discovering new species, fossils, minerals etc.

Nice project. A low DA capacitor Cf in parallel with the gain setting resistor Rf can average out more noise if needed. It can even deal with the mains noise if that remains an issue. Another potential improvement: addition of an optical chopper wheel to deal with the entire system's 1/f noise. The downside is this would require higher sampling rate which would then get demodulated + filtered externally. Would also limit Cf to higher frequencies.

Similar project with single photo diode enclosed in sort of a pinhole camera: https://hackaday.com/2016/05/18/using-missile-tech-to-see-li...

Between this and the DIY radio telescopes and wifi radar, maybe we will have a real tricorder some day.

I'm just happy with all these "Amateur Scientist" links lately. More of these!

hehe no mention of the new iphone 15

To save time waiting for the content to animate in:

  javascript:document.querySelectorAll(".animate-box").forEach(e => { e.classList.remove('animate-box') })

This is very nice, but most of that $10k probably pays for certification that you won't get for DIY. Great for hobbyist, but you couldn't sell it for $9500 profit.

Possibly, but he also describes pretty cheap and easy calibration methods. It's possible that most of the cost of the commercial options is the fact that these are extremely low-volume devices and the overhead of a business with low-volume sales is quite high. You might be paying less for calibration and more for all the things a business needs to do/have that a lone DIYer doesn't.

Not to mention the fact that a lot of researchers are buying thing with grant money, and so can be, in some cases, somewhat price insensitive.

I don't doubt that the commercial one is pre-calibrated and certified. But I would be quite surprised if that certification/calibration _actually_ cost ~$9000.

Sure I'm not saying that. If you give away the R&D for free and don't seek profit you can build it much cheaper.

But self-calibrating is still something else. Unless I pay some lab with traceable calibration to do it for me I can't use the results to certify other equipment for example. I think it's more like insurance, sure the unit cost is low but you pay extra so in case the lab screws up they pay for the mess.

That's fair, but I think a lot of people other than "DIY hobbyists" don't have a need for traceable, liability responsible, validation chains.

DIY is not just for hobbyists.

Companies face the build-or-buy question every day. This article demonstrates that you can build and validate your build with the knowledge and skills expected of most principal investigators. If I were a lab director, I would want my people to be able to consider building when it makes sense, particularly since we can then integrate systems and tailor validation to our requirements.

That's the stuff of proprietary IP and career advancement.

The point wasn't "you can get rich selling this", but "you can save $$$ building this yourself as a hobbyist".

Absolutely, you are correct.

I'm not sure about certification but there is a lot of overhead in designing, producing, selling and servicing products. This is basically a prototype. It will probably work for a while but will be difficult to maintain if something goes wrong and the software won't be very polished. There are a lot of non-BOM costs that go into selling a commercial spectrometer. This is great if you want to DIY one but even a single sensor based visible light spectrometer with cheap parts can't be bought for close to the sum of its parts.

If you're using it in a lab, you can calibrate and verify this sort of instrument yourself most of the time. Wavelength is trivial to calibrate. Absolute amplitude not so much, but you usually don't care that much about absolute amplitude. Response flatness across the spectrum may or may not be a concern.

So what? If you need it, now you can build it. As a hobbyist you don't need certified gear, you need gear.

I wonder what 3rd party certification would cost. Industries that use sensors like this frequently require annual recertification, so there are typically multiple testing houses and competitive prices.

There is a huge difference between calibration and certification. Calibration means that your instrument gives you absolute rather than relative output. Certification means that your instrument is precise enough to be used for specific procedures.

Calibration can be done in house, certification is usually the domain of some certification institution and can be extremely expensive depending on the kind of gear.

DIY recertification obviously. What's the lowest cost for the hobbyist ? A high-end, stable, double-wall PH-meter or finding a cheap way to build single-wall PH meters ?

https://www.instructables.com/cheap-DIY-electronic-pH-meter/

You explore the solution-space, and suddenly, out of nowhere it changes your perspective on another aspect, and you realize you may not need a spectrometer at all to measure ion concentartion in aeroponic nutrient solutions.

http://jupiter.plymouth.edu/~jsduncan/courses../2011_Fall/In...

My first thought as well. People tend to misunderstand why things cost what they do, for stuff like this it isn't the raw components.

I wonder if this could be used for amateur astronomy? Would be cool to point a telescope at a star, and print out a spectrum corresponding to elements with emission lines.

For that a prism and a photographic film would probably be sufficient, the developed film would serve as your print-out.

https://petapixel.com/2019/07/13/shooting-high-res-thermal-p...

One of my favorite photoshoots merges IR and visible.

I think this design is usually called a "scanning monochromator" - really nicely done.

Very cool! NIR is often used to measure protein, fat, water and fiber contents i corn, animal feedstuff. I used to make calibrations for that in the 90's. I wonder if this would be able to do that?

This is pretty cool. One of the more fun things in O Chem lab was using the IR Spec on liquids. Preparing solids for IR analysis was not fun.

I don't understand how it scans to record the spectrum with a point detector? did I miss something?

oh n/m it has a lead screw and moves it across the focal plane of the spectrometer..

Great content and form. The site is clean, yet still uses tasteful transitions and typography.

A different design could leverage laser micro ablations on glass over a traditional CMOS sensor.

Is nobody going to address why copper winding wire was used for DC power and signaling?

Articles like these are rare, but there's 30 years of them scattered around the internet. There's got to be a way to catalogue them in a single index somewhere. Like a Wiki, but just articles that are hard to find and extremely interesting.

I'd like an index because HN's articles are often not this caliber of "interesting". If I look back 1 month to the top of HN (https://news.ycombinator.com/front?day=2023-08-12), there's pontifi-posting (editorials), news and editorials from large media companies, blogspam from tech companies and OSS projects, and basic tech How Tos. I don't want to read any of that; I come back hoping there's a single one of this kind of article, and maybe find one a month.

I have to agree with you. While I do enjoy the OSS projects/physics-/math-related topics (and a lot of the Show HN), these are my absolute favorite things to read.

Curious if you've found other resources that provide more of this sort of content?

I've found the Hackaday blog[0] tends to have a lot of this kind of content (often summarized/linked to the original source) and sometimes it's tagged in a way that makes surfacing others from their archives possible, but I've not found any other sites that are reasonably organized to help surface write-ups of this kind[1].

[0] https://hackaday.com/blog

[1] There are subreddits where this sort of thing can be found, but "deep-dive but accessible technical articles" tend to be placed in a sub-reddit that's dedicated to more narrow topics and a lot are lower quality (there's similar content to this in /r/Optics from 2022, but I'd never have a reason to end up there)

Manually curated list could be crowdsourced via an OSS project, e.g.

  1. Criteria for articles in list.
  2. Example and anti-example articles.
  3. Tag via HN comment with short, unique and human-meaningful phrase.
  4. Query Algolia periodically, triage, PR submission/review/merge.
  5. Syndicate list as RSS feed.

Sounds like you want a curated feed. There are some email mailing lists that aim to achieve this, usually for a specific domain (like stock market, or AI, etc)

Maybe a group chat or discord of like minded people who casually share things they come across that pass a high standard

One way or another it seems like you're going to need to rely on another person to sift through all the daily published content and mark potentially interesting ones

I wonder if you could just feed HN titles with more than ~30 upvotes to an LLM and ask it to bring out similar topics. Not perfect, but I think it could work as a start.

I have one of these[1] Spectruino spectrometers laying around, and I've occasionally used it for fun projects over the years, most recently in a "jar of pond water" experiment, wherein I attempt to keep a glass of pond water alive so that it grows its own ecosystem[2].

I had some success with this pond water experiment, such that I was able to observe multiple Hydra generations feasting[3] on the fruits of my labors.

I mostly used the Spectruino for 'marvel' factor, as in .. was able to see the spectral signature of the jar experiment change over time. I regret that I don't have my daily spectral log images available at the moment, but its something I always wanted to get back to .. one of these days ..

[1] - https://www.myspectral.com/project/spectruino-three/

[2] - https://imgur.com/user/waterplusalgaeequalshydra

[3] - https://imgur.com/gallery/e5WLDqc

Hey nice Atmos and Microdisc btw :)

Yup! Great machine!

Reply on news.ycombinator.com