Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

This situation (lack of schematics) is powering the open source test instrument surge.

Test instruments are an interesting space in the technology center mostly because they are never "obsolete." If they are operating to their original specifications and tolerances, they do the job just as well in 2020 as they did in 1960. As a result, test equipment can really hold its value well.

I recently paid about $700 to have my HP Spectrum Analyzer from the early 90's[1] calibrated. I bought the instrument for $1,100 at auction, it's working range is 9kHz to 12GHz. So for $1,800 I have an instrument that, if I were to buy the current generation model, would cost me nearly $18,000 or 10x that price. Even a "cheap" Rigol that only goes to 6.5GHz would cost $10,000.

So what to do when your sales are "one and done" ? It is a hard problem from a business model perspective.

[1] EDIT: Turns out it is only 30+ years old, not 50+. Its an HP8596E "portable" (weighs quite a bit :-).

View on HN

Here's where I have to chime-in with a reality that people have yet to experience and most don't know about: The unintended negative consequences of RoHS.

I have over a dozen instruments and several tools that are, in some cases, over 30 years old. I think all of my scopes, signal generators, DVM's, logic analyzers, DSO's, probes, lab power supplies, etc. are pre-RoHS and, in most cases, significantly so.

The transition to RoHS, while, in principle, well-intentioned, is likely to prove to have been a massive mistake.

Lead-free solder has one major problem: Tin whiskers.

One way to think about this is that all RoHS electronics has a stochastic failure rate. I have devoted more time than I care to admit studying tin whiskers in the context of my work in aerospace. I have, in that process, consulted with NASA scientists who were are the forefront of long term research on the subject. The most salient take-away was that we had no way to predict or truly mitigate tin whiskers. The only mitigation in aerospace is to, quite literally, send chips to services that remove all lead-free solder from the pins (or remove balls from BGA's) and replace them with conventional lead solder.

If this plays out as it could, landfills are going to be piled sky-high with broken electronics. From phones to laptops, TV's, ovens, clocks...anything really. And cars, yes, cars!

Lead-free RoHS solder is a ticking time bomb and, in my opinion, one of the most misplaced decisions made in the name of protecting the environment.

I have HP-41 calculators I bought in the early '80's that still work as new today. That's to say they are nearly 40 years old. There is no way a RoHS compliant calculator will survive 40 years. That is nearly impossible. And so, millions of them will end-up in landfills. Well done European Union, you really helped the planet with that one!

For those not familiar with RoHS issues (a deep and wide topic), here's a starting point:

https://nepp.nasa.gov/whisker/reference/tech_papers/2006-Lei...

This is very real.

TIL: RoHS == Reduction on Hazardous Substances.

https://en.wikipedia.org/wiki/Restriction_of_Hazardous_Subst...

Congratulations you're part of today's 10,000. RoHS is required in most European markets and most electronic musical equipment since ~2005 is RoHS

I agree, and my belief is that RoHS is a convenient excuse for companies to continue creating e-waste, as long as it's "environmentally friendly" e-waste.

I suspect a similar story with "biodegradable" materials too, they just haven't become common enough to workd their way into products which people expect to last longer.

Also the fact that some critical categories are exempt (military, aerospace, automotive, active implantable medical devices etc.) speaks volumes about that.

The lead problem is with e-waste, basically manually disassembling/melting electronics for the parts, not while the device is in use.

If I remember history, it was France who pushed for RoHS. Manufacturers wanted nothing to do with it precisely due to reliability issues. When the EU made it a requirement the entire world was forced to follow.

I remember having to retool most of my products at the time. It was nothing less than a nightmare.

Just thought of this while reading your post: Could the tin whisker problem be solved by simply reflowing circuit boards every 5-10 years, before they actually short out? I mean, it's labour intensive but for specialized or hobby equipment, might be worth doing.

Most complex boards will have a few parts (like plastic connectors) selectively soldered on after reflow.

True. I wonder if you could do something with IR and a mask to selectively heat the bits of the board that’d take it?

Then again, later in the slides they talk about simply giving things a scrub with a wire brush to clean off the whiskers. Apparently they don’t regrow after the first few years.

Theoretically - yes. Practically - just imagine the logistics of that.

> Could the tin whisker problem be solved by simply reflowing circuit boards every 5-10 years

Apart from logistics, many components have a hard time surviving the heat of soldering during original manufacture and may fail on subsequent reflowing.

Lead free solder typically has a melting point around 35C higher than tin-lead which exacerbates the problem.

I had heard about tin whiskers, but always in relation to NASA. I had assumed it was only a problem in space, somehow related to lack of gravity and/or atmosphere.

Apparently it's just that NASA cares about reliability more than most, and hence takes the time to do more detailed failure analyses.

Thanks for linking those slides.

NASA has done tons of research on this over decades. I have read most of the papers they published and worked with a couple of their researchers. While the context was aerospace, the effect happens on earth just as well. Pretty much all of their tests are done on the ground.

The reason lead was added to solder in the ~30s-40s was tin whiskers. Think about how huge components and pitches were back then and they still noticed tin whiskers and changed the solder to avoid them.

Repairing a satellite in flight is effectively impossible. Thus NASA cares far more about how long something will last than most users.

In 1992 I got a Swatch "Earth Day" edition self winder. Made from the best modern earth friendly materials. By 95 it had cracked and yellowed to the point it was illegible.

I started wearing my grandfather's '68 Caravelle self winder. I still am.

If I remember correctly Swatch had horrible problems with tin whiskers.

Here’s an interesting article from 2008. Among other things, it discusses the failure of a nuclear power plant due to tin whiskers.

http://www.sigcon.com/Pubs/news/10_01.htm

Ideally, we would be building devices that don't suffer from tin whiskers in the long term. Pragmatically, the vast majority of e-waste is being generated from simpler failures than tin whiskers (degraded batteries, simple mechanical breakage, changing standards and performance requirements, etc). In that case, it is better that the supply chains for those products are lead-free, reducing harm in production and in disposal.

We need to make our consumer products longer lived and circular, and that should be a prerequisite for RoHS exemptions (as it is currently for certain classes of long lived non-consumer devices).

Funny, the simplest thing to do there, and relatively cheap, is to lacquer the whole device after it's soldered. This is done on military grade hardware - or even tougher conformal coating.

Would that help against lead, or against tin whiskers? AFAIK tin whiskers can develop through lacquer too. But probably it helps some.

> Funny, the simplest thing to do there, and relatively cheap, is to lacquer the whole device after it's soldered. This is done on military grade hardware - or even tougher conformal coating.

That helps avoid tin whiskers, but those aren't the only problem with lead free solder. It doesn't flow as easily as tin-lead, so dry joints are more common. It's relatively brittle, so it's more susceptible to vibration and thermal fatigue. Melting point is higher so components are more likely to be damaged during soldering.

Military hardware is exempt from RoHS for good reason.

Sadly, no, this does not avoid or stop tin whiskers at all. The general technique you are referring to us called “conformal coating”. It can help a bit but it does not stop growth. Also, at that scale the whiskers are super strong and easily puncture right through the coating.

This is a deceptively complex topic (as I learned over the years). For example, a whisker that does not penetrate the coating will either buckle and curl-up under the film or grow laterally under it. With fine pitch components having pin-to-pin gaps in the range of 0.25 mm, a short between pins can happen in just a few weeks.

Conformal coating is useful for other reasons, but, ultimately, it isn’t a solution for tin whiskers.

Back in the day I even looked into using specially formulated epoxy encasement. What starts happening there is that the coefficient of thermal expansion differential between The epoxy, board and components can cause all kinds of failures...and it still does not stop tin whisker growth or buckling.

This is fascinating, I'd never heard of tin whiskers before today. From a short look at the literature, nickel-palladium-gold alloys seem to be an alternative lead-free finish that doesn't suffer from whiskers. Have you worked with that at all? Does it have other shortcomings?

Well, being that most of my work during the last few years has been in aerospace we just use conventional lead-containing solder. When you can't afford for things to fall out of the sky you go with what you know works. We've known about tin whiskers since, I think, the 1940's. In fact, if I remember the history correctly, we added lead to solder precisely for this reason way back in the prehistoric era.

I might be diving into some industrial work soon that does not benefit from RoHS exemptions in this domain. This is one of the important cards in my kanban board...likely a rabbit hole that will not be pleasant to navigate.

I took a quick look myself. Here's a note from Maxim with interesting data:

https://www.maximintegrated.com/en/design/technical-document...

I've used pretty much all the conformal coatings they list in this article. What they don't go into is buckling under the coating, which can short adjacent pins in fine pitch chips.

I truly hate this problem. I has wasted more of my time over the years than anything else. RoHS is, in my opinion, a very misguided directive. It's a perfect example of when politics gets ahead of reason and fear-mongering wins over science. If we are not careful we are on track to do something similar with climate change.

> the vast majority of e-waste is being generated from simpler failures than tin whiskers

I can't assert or challenge this conclusion because we simply do not have the data. This is yet another reality of this issue: Tin whisker failures in the consumer domain are nearly 100% unreported.

Consumers are certainly not in a position to conduct the forensic work required in order to determine failure modes. My guess is that the large majority of devices are thrown out upon failure, which means we know nothing about why they failed.

Companies, on the other hand, frankly, have little incentive to conduct that research other than to use best practices during design. If a product survives a couple of years they are good. This isn't due to some dark profit motive, it's just a reality of business. You simply can't ask every single customer to send you their failed hardware for expensive --very expensive-- forensic analysis no matter their age or condition. This would mean having to manage a process involving forensics of millions of devices per month or year, depending on scale.

When I was investigating tin whisker mitigation options I explicitly sought out data from the consumer electronics domain. I quickly discovered there was none, at least nothing useful. This is why I don't think a statement such as the one you made is neither right or wrong, it simply has virtually no supporting data in either direction.

I should rephrase. From my experience in consumer electronics, the vast majority of instances of consumers stopping using a device are not from unknown failures. They are from visible failures (including reduced battery life), software/content incompatibility or obsolescence, or the consumer just generally losing interest in the device. I should caveat though that this is in PC and mobile categories. It’s plausible that things like appliances don’t fall into those failure or obsolescence modes and could be facing unknown failures that are solder related.

Yeah, it's hard to quantify in any meaningful way. That's why I said I am not in a position to confirm or challenge what you said and all I know is that, for the most part, we can't really make scientifically supportable statement about the consumer world (by this I mean with actual published data from an authoritative source). We are blind.

Yes, lead-free solder brings a whole lot of trouble to the table. Perhaps someday someone will come up with a formula that doesn't -- but until then, I continue to use lead-based solder.

> So what to do when your sales are "one and done" ? It is a hard problem from a business model perspective.

The vendor's responses since the ~90s is pretty much that they do trade-ins and crush the old gear. That is one of the reasons (the other being outsourcing) why test instruments made in the last ~20-25 years are much rarer in EU/US compared to older gear.

Make instruments and sell them like furniture makers, carpenters, most kinds of law (how many wills do you need?) etc

Sort of a 'make on demand' model. It might work. These days it is hard to find parts that don't get end-of-lifed after 5 years, so you're re-doing schematics and thus your instrument's PCB. You could perhaps make a modular structure such that the instrument was constructed from modules but I'm not sure how well that would work.

The open source trend is to put the instrument in its box and connect via USB to a computer that provides the UX. That works but has the issue of computer operating system compatibility. I've got a USB scope in my junk drawer that has a UI that only works on Windows 98. Not too useful these days, and the company is out of business.

The instrument control protocol standards help here.

To some extent standardized assemblies have been used for a very long time. HP System 2 (the cute 1/3 rack units) is from the 60s, System 3 from the late 70s (I think), though the actually walked back on that one and later replaced the standardized cast aluminium parts with unit-specific sheet metal bending (3478A and others). Later they re-used the display/input modules, both HP and Keithley did this for a lot of their later DMMs and SMUs.

I assume that the program that only ran on Win 98 is not open source. If it was, chances are someone would have ported it to Linux by now.

You would be correct. Of course it is just a USB 2.1 box and enumerates as a HID device and a bulk data endpoint so presumably spending some time with a USB protocol analyzer and running it through its paces would illuminate some aspects of its operation.

This is why its still in my junk box and not in an ewaste skip somewhere :-)

What model is it? There's a non-zero chance that someone has done that work already and made it work via sigrok, https://sigrok.org/wiki/Supported_hardware#Oscilloscopes

-Perhaps a stupid question, but wouldn't that UI live and thrive inside a Win98 VM?

(I am cursed by the instrumentation gods - I have to keep all sorts of late-paleolithic configuration software running to support the ditto hardware we still rely on for specific tasks in our service department.

I've found that just about anything can be coaxed into working - at least as long as it doesn't involve some copy protection scheme relying on slightly iffy RS232 timing and a dongle.

> This situation (lack of schematics) is powering the open source test instrument surge.

Actually, I would argue that the expense of tech support is what is powering the open source test instrument surge.

Unless I can sell you a $20,000 instrument and cut you off after 3 years, my support costs are probably unsustainable.

This means that anything worth less than $20,000 is unprofitable (case in point: the Tektronix USB Real Time Spectrum Analyzers). So, the only way around that is to make it worth even LESS and make it open source.

Now, your tech support is the internet at large. Good luck with that.

Sure when you can't find connectors, power cables, comparable software or accessories. You got your instrument compared to a traceable reference but good luck getting an adjustment or repair performed. This only applies to dumb instruments (power supplies and dc electronic loads) but remember the technology you are measuring is eventuality going to have performance that exceeds your ability to test.

What is this "software" you speak of?

Seriously, I strongly prefer test equipment from the pre-software era. Test equipment shouldn't have to boot, and it shouldn't be dependent for its operation on the shitty non-realtime junk we call operating systems nowadays.

As do I. That said, there is something to be said for the economics of using software to compensate for manufacturing tolerances in hardware. 3D printers are a good example of taking a crappy platform and making it perform reasonably precise things using clever software.

Of course sometimes that software is in an FPGA design :-). I expect that once the Xilinx RFSOC starts ending up in test gear you'll see some more cost effective GHz gear but very unhackable due to the FPGA nature of things.

Of course no amount of software is going to make the front end conditioning circuits quieter (well I suppose simulating them and designing from that might but not on the instrument itself). That will always need a certain expertise to design and implement.

I dunno, I've been doing a lot of RF analysis lately as my company has been re-upping our capabilities in that regard. We went from an 80's spectrum analyzer (the Rohde and Schwarz rep laughed at us when we called to ask for calibration data... "that machine was made before you were born") hooked up to an ink plotter to a mid-level Rigol analyzer. It really made me appreciate being able to recall the settings for the scan I ran last week, dump the data for the new scan, baselines and a few variations I wanted to try, and compare them all in Excel back at my desk (alongside the results from the external compliance lab).

Fair point. Sending data back and forth is certainly useful. My main gripe is with "instruments" that are themselves just applications on a PC with a little bit of external A/D interface hardware. I prefer standalone boxes, perhaps with a data interface but not dependent on it.

The newest Keysight spectrum analyzers all run Windows 10. Even an older Tektronix scope I use daily runs XP. The XP install has crapped out a few times but it has a recovery partition built in so it's trivial to restore it.

Guessing a 141T ?

Good guess, but my date was wrong. It wasn't 1974 it was in the early 90's apparently (8596E btw). I suspect there is a decoder ring for the serial number which would tell me exactly. I've updated the date in my original comment.

The first two digits of the serial number gives the years since 1960 of the revision for HP.

Hmm, well then it would be 1999, first two digits are 39.

i would expect any HP RF test equipment from any age to be fixable and usable (worse case, perhaps new vacuum tubes!)

Can’t beat 20th century HP test equipment. The pinnacle of quality.

I have an 8566B at home. These things are built like tanks, and were made to operate in high temperature environments. We had about 10 of them at work, and the only things that would fail (slowly) are the CRT supplies. I’ll get the LCD retrofit when that happens.

Have used one of those ! The 'automation' for its time was great. And calibration; signal amplitude measurements were always right on.

Why are they so expensive?

Maybe because they've figured out they're "one and done"?

Because they're incredibly complex, need to be long-term reliable and are made in small-ish numbers, sold to a small professional customer base.

Broadly speaking test instruments often do similar things that some consumer devices do as well, but 1) with greater precision, accuracy and stability 2) over a much wider range of parameters.

For example, a spectrum analyzer is similar to a radio receiver, but the SA usually covers a much larger rage (typically 9 kHz - 1.8, 5, 12 GHz), the SA can handle a huge range of input powers (typically something like -1xx to +20 dBm), and it can accurately measure power levels at all frequencies, has a variety of filters and analysis functions and so on.

The RF section of some of these is actually mostly made of standard COTS parts - input assembly (these use very high grade connectors because of the repeatability requirements), programmable precision attenuator, mixer module, YIG LO etc. -- except all of these are high precision, RF devices, which simply cost a lot of money.

The 80/20 rule applies as well. Well, maybe more like 60/40. An SDR can get you most of the stuff a proper SA can do for a fraction of the price (and can do some things an used SA from the 80s or 90s cannot do). A very good audio interface can serve as an audio analyzer, given the right software. A common pattern is that you can get the capabilities to do something, but not the accuracy of a real test instrument.

One aspect that drives cost is that high precision circuits often need a few parts that have tighter tolerances than the typical 5% or 1% that are usually good enough for 99% of use cases. And these parts are a lot more expensive.

It's worth mentioning that often parts aren't manufactured to be high precision variants. That would be extremely expensive. What happens instead is the parts get measured and "binned" by how close to spec they are. In a large enough sample, statistically a few parts (even with an inaccurate process) will be perfect. The best ones are sold as A grade, etc.

This is pretty common for thinfs like resistors, but even CPUs and RAM are binned by their overclock performance as there's some randomness in production.

When I were a wee lad, we used to machine passive components to value (sometimes the "whole thing", sometimes a companion trim piece). There's no economy of scale to be had there, and binning won't help all that much because it's the installed value that matters, not the loose part value.

Reply on news.ycombinator.com