Where Have All the Schematics Gone?
groups.io
groups.io
Test instruments are an interesting space in the technology center mostly because they are never "obsolete." If they are operating to their original specifications and tolerances, they do the job just as well in 2020 as they did in 1960. As a result, test equipment can really hold its value well.
I recently paid about $700 to have my HP Spectrum Analyzer from the early 90's[1] calibrated. I bought the instrument for $1,100 at auction, it's working range is 9kHz to 12GHz. So for $1,800 I have an instrument that, if I were to buy the current generation model, would cost me nearly $18,000 or 10x that price. Even a "cheap" Rigol that only goes to 6.5GHz would cost $10,000.
So what to do when your sales are "one and done" ? It is a hard problem from a business model perspective.
[1] EDIT: Turns out it is only 30+ years old, not 50+. Its an HP8596E "portable" (weighs quite a bit :-).
The open source trend is to put the instrument in its box and connect via USB to a computer that provides the UX. That works but has the issue of computer operating system compatibility. I've got a USB scope in my junk drawer that has a UI that only works on Windows 98. Not too useful these days, and the company is out of business.
The instrument control protocol standards help here.
This is why its still in my junk box and not in an ewaste skip somewhere :-)
(I am cursed by the instrumentation gods - I have to keep all sorts of late-paleolithic configuration software running to support the ditto hardware we still rely on for specific tasks in our service department.
I've found that just about anything can be coaxed into working - at least as long as it doesn't involve some copy protection scheme relying on slightly iffy RS232 timing and a dongle.
I have an 8566B at home. These things are built like tanks, and were made to operate in high temperature environments. We had about 10 of them at work, and the only things that would fail (slowly) are the CRT supplies. I’ll get the LCD retrofit when that happens.
For example, a spectrum analyzer is similar to a radio receiver, but the SA usually covers a much larger rage (typically 9 kHz - 1.8, 5, 12 GHz), the SA can handle a huge range of input powers (typically something like -1xx to +20 dBm), and it can accurately measure power levels at all frequencies, has a variety of filters and analysis functions and so on.
The RF section of some of these is actually mostly made of standard COTS parts - input assembly (these use very high grade connectors because of the repeatability requirements), programmable precision attenuator, mixer module, YIG LO etc. -- except all of these are high precision, RF devices, which simply cost a lot of money.
The 80/20 rule applies as well. Well, maybe more like 60/40. An SDR can get you most of the stuff a proper SA can do for a fraction of the price (and can do some things an used SA from the 80s or 90s cannot do). A very good audio interface can serve as an audio analyzer, given the right software. A common pattern is that you can get the capabilities to do something, but not the accuracy of a real test instrument.
This is pretty common for thinfs like resistors, but even CPUs and RAM are binned by their overclock performance as there's some randomness in production.
The vendor's responses since the ~90s is pretty much that they do trade-ins and crush the old gear. That is one of the reasons (the other being outsourcing) why test instruments made in the last ~20-25 years are much rarer in EU/US compared to older gear.
Seriously, I strongly prefer test equipment from the pre-software era. Test equipment shouldn't have to boot, and it shouldn't be dependent for its operation on the shitty non-realtime junk we call operating systems nowadays.
Of course sometimes that software is in an FPGA design :-). I expect that once the Xilinx RFSOC starts ending up in test gear you'll see some more cost effective GHz gear but very unhackable due to the FPGA nature of things.
Of course no amount of software is going to make the front end conditioning circuits quieter (well I suppose simulating them and designing from that might but not on the instrument itself). That will always need a certain expertise to design and implement.
I have over a dozen instruments and several tools that are, in some cases, over 30 years old. I think all of my scopes, signal generators, DVM's, logic analyzers, DSO's, probes, lab power supplies, etc. are pre-RoHS and, in most cases, significantly so.
The transition to RoHS, while, in principle, well-intentioned, is likely to prove to have been a massive mistake.
Lead-free solder has one major problem: Tin whiskers.
One way to think about this is that all RoHS electronics has a stochastic failure rate. I have devoted more time than I care to admit studying tin whiskers in the context of my work in aerospace. I have, in that process, consulted with NASA scientists who were are the forefront of long term research on the subject. The most salient take-away was that we had no way to predict or truly mitigate tin whiskers. The only mitigation in aerospace is to, quite literally, send chips to services that remove all lead-free solder from the pins (or remove balls from BGA's) and replace them with conventional lead solder.
If this plays out as it could, landfills are going to be piled sky-high with broken electronics. From phones to laptops, TV's, ovens, clocks...anything really. And cars, yes, cars!
Lead-free RoHS solder is a ticking time bomb and, in my opinion, one of the most misplaced decisions made in the name of protecting the environment.
I have HP-41 calculators I bought in the early '80's that still work as new today. That's to say they are nearly 40 years old. There is no way a RoHS compliant calculator will survive 40 years. That is nearly impossible. And so, millions of them will end-up in landfills. Well done European Union, you really helped the planet with that one!
For those not familiar with RoHS issues (a deep and wide topic), here's a starting point:
https://nepp.nasa.gov/whisker/reference/tech_papers/2006-Lei...
This is very real.
https://en.wikipedia.org/wiki/Restriction_of_Hazardous_Subst...
I suspect a similar story with "biodegradable" materials too, they just haven't become common enough to workd their way into products which people expect to last longer.
I remember having to retool most of my products at the time. It was nothing less than a nightmare.
Then again, later in the slides they talk about simply giving things a scrub with a wire brush to clean off the whiskers. Apparently they don’t regrow after the first few years.
Apart from logistics, many components have a hard time surviving the heat of soldering during original manufacture and may fail on subsequent reflowing.
Lead free solder typically has a melting point around 35C higher than tin-lead which exacerbates the problem.
Apparently it's just that NASA cares about reliability more than most, and hence takes the time to do more detailed failure analyses.
Thanks for linking those slides.
I started wearing my grandfather's '68 Caravelle self winder. I still am.
Here’s an interesting article from 2008. Among other things, it discusses the failure of a nuclear power plant due to tin whiskers.
We need to make our consumer products longer lived and circular, and that should be a prerequisite for RoHS exemptions (as it is currently for certain classes of long lived non-consumer devices).
That helps avoid tin whiskers, but those aren't the only problem with lead free solder. It doesn't flow as easily as tin-lead, so dry joints are more common. It's relatively brittle, so it's more susceptible to vibration and thermal fatigue. Melting point is higher so components are more likely to be damaged during soldering.
Military hardware is exempt from RoHS for good reason.
This is a deceptively complex topic (as I learned over the years). For example, a whisker that does not penetrate the coating will either buckle and curl-up under the film or grow laterally under it. With fine pitch components having pin-to-pin gaps in the range of 0.25 mm, a short between pins can happen in just a few weeks.
Conformal coating is useful for other reasons, but, ultimately, it isn’t a solution for tin whiskers.
Back in the day I even looked into using specially formulated epoxy encasement. What starts happening there is that the coefficient of thermal expansion differential between The epoxy, board and components can cause all kinds of failures...and it still does not stop tin whisker growth or buckling.
I might be diving into some industrial work soon that does not benefit from RoHS exemptions in this domain. This is one of the important cards in my kanban board...likely a rabbit hole that will not be pleasant to navigate.
I took a quick look myself. Here's a note from Maxim with interesting data:
https://www.maximintegrated.com/en/design/technical-document...
I've used pretty much all the conformal coatings they list in this article. What they don't go into is buckling under the coating, which can short adjacent pins in fine pitch chips.
I truly hate this problem. I has wasted more of my time over the years than anything else. RoHS is, in my opinion, a very misguided directive. It's a perfect example of when politics gets ahead of reason and fear-mongering wins over science. If we are not careful we are on track to do something similar with climate change.
I can't assert or challenge this conclusion because we simply do not have the data. This is yet another reality of this issue: Tin whisker failures in the consumer domain are nearly 100% unreported.
Consumers are certainly not in a position to conduct the forensic work required in order to determine failure modes. My guess is that the large majority of devices are thrown out upon failure, which means we know nothing about why they failed.
Companies, on the other hand, frankly, have little incentive to conduct that research other than to use best practices during design. If a product survives a couple of years they are good. This isn't due to some dark profit motive, it's just a reality of business. You simply can't ask every single customer to send you their failed hardware for expensive --very expensive-- forensic analysis no matter their age or condition. This would mean having to manage a process involving forensics of millions of devices per month or year, depending on scale.
When I was investigating tin whisker mitigation options I explicitly sought out data from the consumer electronics domain. I quickly discovered there was none, at least nothing useful. This is why I don't think a statement such as the one you made is neither right or wrong, it simply has virtually no supporting data in either direction.
Actually, I would argue that the expense of tech support is what is powering the open source test instrument surge.
Unless I can sell you a $20,000 instrument and cut you off after 3 years, my support costs are probably unsustainable.
This means that anything worth less than $20,000 is unprofitable (case in point: the Tektronix USB Real Time Spectrum Analyzers). So, the only way around that is to make it worth even LESS and make it open source.
Now, your tech support is the internet at large. Good luck with that.
Company B releases a product that works pretty much as well as Company A's but is cheaper (because it isn't investing in the aftersales support). Since support is an expensive business, by cutting it Company B makes more profit.
In the long run, Company B wins. Quite a few companies are smart enough to see how this works, and so morph from being an A type company to being a B. Evolution in action, the fitter (most profitable) company survives (shame about the prey..ah, "clients").
Valuing up-front over TCO only makes sense for very large purchases and for business investments.
In the case of Verizon, signing up with such a plan also allows you to return and upgrade your phone after it's halfway paid off.
With phones getting more and more ridiculously expensive, there's little reason not to do this unless you'd rather be out $1000+ immediately when you have the option to be out $40/month for a couple years with no downsides instead.
Which many, many, many... do.
The world's largest retailer is Wal-mart for a reason (which has nothing to do with TCO and everything to do with up-front price of goods).
Hence this is a prime example where regulation would be needed.
No, it shows that markets work perfectly - they just optimize for what the customer wants to buy, not for some noble goal that you desire.
The customer wants machines that are more difficult to maintain and repair?
Is the customer a masochist?
On the other hand, if the noble ethical goals that you list were more important for the customer, he would prefer devices that satisfy these goals (as much as possible) over low prices in his buying decision(s).
See e.g. the right-to-repair debate. The function is important enough for farmers to demand regulation and drive the prices of old, non-locked-down tractors through the roof. And yet, it's still more profitable for John Deere to lock down their tractors.
When deciding between the packages "modern, feature-rich tractors that are locked down" or "old, reparable but overpriced tractors", the former probably win. That still doesn't imply that farmers "want" locked down tractors or that reparability is of no importance to them.
Also yes, we do need ethics in business. Last thing I've read, even some MBAs have heard of that by now.
This is in fact a textbook example for how markets work. By the insane prize increase of old, non-locked-down tractors, every sane market participant can see how much more of a prize farmers are willing to pay for non-locked-down tractors.
I believe every manufacturer of tractors now does a careful market analysis and is probably (secretly) developing a more open, but expensively prized tractor.
My bet: In a few years there will be new tractors on the market that are better repairable, but higher prized - exactly what an economist would predict. It's just that we are currently in a transformation period.
In case of tractors it's not at all obvious how fast the market will provide the tractors with lower TCO, that farmers wish for.
Information scarcity is a defining feature of modern market, let's not forget that.
What you ignore is that there are a bunch of information and experience asymmetries here, some of them intentionally created by vendors. There are also big problems with short-term biases and principal-agent problems. These lead to demonstrably sub-optimal global outcomes.
As an example, look at the recent wave of Softbank-funded idiocy. There's no way WeWork was an optimal deployment of capital. You could have lit $10 billion on fire and done better. But it worked out very well for Adam Neumann, who not only got lots of short-term adulation and wealth, but somehow ended up becoming a billionaire because they paid him to go away. And this is from the supposedly hyper-rational experts in the world of finance. If they can't get it right, there's little reason to think that random purchasers and Tektronic's middle managers will.
Right, and it delivers that by pushing costs onto the greater proportion of people who are not involved. That is neither ethical nor sustainable -- it's more like a kind of theft.
What you are citing is a clear example of how the "free market" can fail to operate in the best interests of society.
The market is intrinsically linked to its effects on the environment it's in and I would certainly consider its failure to account for that a failure. To put it in CS terms, if build a "world peace" gAI with a reward function that doesn't include "not killing humans" and it starts killing humans, the argument of "it's working perfectly" isn't really a useful one if you want to stay alive. Adding a negative term to the reward function is equivalent to marker regulation, yet while we have entire fields of study dedicated to the first, the latter is frowned upon.
Regulation is mandatory in order to have a society. The question isn't whether or not it should exist, but whether or not the particular regulation is good.
For the Fluke, you can, today, get the schematics online in the user manual (2). For the Tek, not so much (3). Regardless, they have outlasted my phones by 15-20x.
Oh, and while I'm on my old electronics worship soapbox, let's mention the HP 48G, one of the greatest calculators ever made, which runs Reverse Polish Lisp (4) and still has an active community (5), 17 years after production ended and 27 years after release, and emulator apps for iPhone (6) and Android (
(1) https://www.radiomuseum.org/r/fluke_digital_multimeter_8060a...
(2) http://assets.fluke.com/manuals/8060a_3vimeng0200.pdf
(3) https://www.arc.ro/userfiles/docs/APPA/multim%20digitale/APP...
(4) https://en.wikipedia.org/wiki/RPL_(programming_language)
See my post on this thread about the "mutually assured destruction" (as I sometimes call it) brought to us by those who pushed for a transition to lead-free solder. This is a real and massive problem that might come home to roost in a major way within the next ten to twenty years. Here's the link:
That said, I doubt anything with a battery will last anywhere close to old tech.
Service manuals from Tektronix, like those from Hewlett-Packard, used to include what amounted to a BSEE lab syllabus in their service manuals. They would include full schematics and operational theory that often went well beyond what was needed by a technician who was tasked with maintaining the gear. Often the documentation was written by the design engineers themselves, who were the Michael Jordans and Kobe Bryants of their field. The educational value of this material was (and is) immense. Not only could you get your scope up and running after spending some time with these manuals, you understood a lot about how it worked and what motivated the underlying design decisions.
Dennis is right in that this wonderful literature went away well before its time, but I think he undersells the other side of the argument, which is that these manuals would inevitably become less useful over time as well. It was true that management became less engineering-driven in the 1980s-1990s period, which was both unfortunate and avoidable, but it's also true that over the same timeframe he's referring to, companies like Tektronix and HP had to migrate to custom ASICs and software-heavy architectures that were inherently less user-serviceable.
These aren't like products from manufacturers like Apple, Tesla, or John Deere whose overt market abuses have spawned the right-to-repair movement -- they're the tools used by the people who design those products. The instrumentation companies had to stay well ahead of the technology curve, just as they do now, and increasing integration often has the unfortunate side effect of making the inner workings of the product less accessible to the user. Personally, I'm not sure things could have evolved much differently than they did, regardless of the ratio of MBAs to EEs in management.
Bummer, too, because those manuals were AWESOME. One thing that is indisputable is that both Tektronix and HP stopped publishing component-level service manuals about 5-10 years before the technology justified it. Because their technology had to stay ahead of the market, a lot of gear that is still very useful in many applications today is now almost impossible to keep running.
I see this trend with automotive manuals too; in particular, from the 40s to 60s when automatic transmissions were still a relatively new and complex part, the service manuals contained a huge amount of information on their theory of operation. Now, although they're even more complex (and computer-controlled), the most you get is how to take it apart and put it back together.
Patents and other intellectual property methods are the only barrier to a competitor reproducing the mechanical product, but software allows additional barriers to reproduction as well as a means to prevent modification of the operation from the manufacturer's desired behaviors.
I'd also note that the era in which those mechanisms were created and documented included an expectation that owners often did their own automotive maintenance and repair and were far more mechanically-minded (it was an age defined by mechanisms) than current owners of autos.
Alas my experience in maintaining many codebases over the years suggest that most code needs a lot more servicing, itself, than the hardware it ran on. ;)
In my view, this is a 100% illegitimate thing to do. Once I own a product, the manufacturer should have no say in what I do to the product.
The reason/motivation was cost. It is very expensive for the consumer to have a subject matter expert in electronics or mechanics do full diagnosis. It's like hiring a programmer to fix your off the shelf software - it isn't cheap and for consumers now that there is a choice, why would they want to pay hundreds of $$$ to replace a cheap cap? It's not the cost of the component that they're charged after all. That too can be applied to cars - after all modern cars are computers on wheels more than they are mechanical. The mechanic only need to be taught how to use a computer to diagnose the car, and be told what part/module needs to be replaced. It's faster and because it's lower skilled it's cheaper per hour too.
This is speaking from someone in the IT field - been there since the beginning of the PC era. We don't really expect code, or make code, to be analyzed line by line. It's modular, full of dependencies and very fluent. To improve code, we often replace whole modules instead of rewriting our code - new libraries, new frameworks is a much faster and cheaper way to improve code, than rewriting it from scratch or attempting to find a way to optimize 10s of thousands of code lines. Only the factory has the experts that design those modules. With computers, the stuff created has a very short life-span - it's often more about "good enough" vs "best/perfect" as tomorrow will have the fixes "automagicly".
As a consequence, the field is much larger than it would be if only the real experts could take part. Imagine the skill set someone doing mainframe programming in the 1940s and 1950ies had to have compared with a programmer today. It's worlds apart. It would be more surprising if other fields hadn't moved towards the same idea. The vast amount of programmers today don't even know how the computer works internally.
I agree with the OP - generally the quality is way down. Does anyone really expect their TV, Blueray player or computer to last for 10 years? You probably already noticed that Blueray isn't really in demand these days - it lasted less than 10 years! The consumer electronics market moves way too fast to be betting on longevity.
Now, speaking as an engineer who likes to tinker with things, like to understand how things work, I do regret this movement. But I have to ask myself if we would have had the progress we've had in electronics if we hadn't focused on making it affordable to the masses. Ask yourself how many TVs would be sold at $8000 instead of Wallmart's $200 made in China, undocumented and often you're unable to find the exact same model after just a few months. But it's cheap - which is why there's a market for it.
So I do not foresee this change back. Not unless the general public gets massively increased purchase power. I'm not surprised the commuter earning $50k/year would prefer this car shop-bill for maintenance isn't in the $1000s but in the low $100s and even that is considered high by many. Recall that in the US more than 50% cannot find enough savings for a $1000 emergency - so it's not buying to save 5 years from now. It's all about the now (even though we all know it's more expensive in the long run). And to lower cost it's modular and "automated" without the need for expensive experts spending hours/days diagnosing to find the exact part that failed.
Why do you think that? What about ASICs and software-heavy makes devices less user-serviceable? I mean, software has schematics you, we usually call those "source code", but it's the same thing, so the assumption would be that devices would come with the source code for the software. With ASICs, I can see a problem when the ASIC itself is broken and you can't buy it anymore, but then I'd think that's probably not the most common failure mode?
ASICS can (and do) break, unfortunately, and schematics continue to be helpful in diagnosing and repairing those sorts of problems. Often an elaborate recalibration process is needed after doing so, requiring all kinds of specialized equipment. As a result, both the supply and the demand for those schematics started ramping down in the 1990s.
Early on in that cycle, some major customers like Boeing continued to insist that schematics be provided, which is the only reason we have some of those documents at all.
But the purpose of a schematic is not to identify a broken component, it's to enable you to understand the system and thus diagnose malfunction, which is exactly what you would need the code for as well. When it's obvious what component is broken (like, a blown cap or a burned resistor or whatever), you don't need a schematic to figure out the fix. You need a schematic to understand all the stuff around the actual defect in order to be able to locate it--and code would be equally useful for that. The fact that there is no wear to repair in the code itself is secondary--there is no (problematic) wear in most of the circuit that you study to find the fault either.
Also, even if there is no wear to repair, there could still be opportunities to modify things either for repair or for diagnostic purposes. Like, make the software produce test signals that you can trace through the circuit. Or "rewire" I/O pins in the software to work around broken hardware. Exactly what you would use schematics for as well: It's not always about replacing a broken part, sometimes you also just modify some other parts of the circuit to restore function or to diagnose a problem, and the same could potentially be done in the software as well.
> ASICS can (and do) break, unfortunately, and schematics continue to be helpful in diagnosing and repairing those sorts of problems.
Well, yeah, sure. But my point is that many defects are in other parts of the circuit, and knowing the internals of the ASIC could still be helpful in diagnosing and fixing those problems.
Which seems unlikely to me, although I wouldn't be surprised either way.
The genius bar is not the problem. It's some things they have done more recently.
I wonder how much of this is due to focus on size/weight gone too far.
Or if Apple is going to tell you how to build a board with function XYZ to their specs, they may as well integrate it into the mainboard.
The idea that it somehow stops competitors from copying the product is very weak; it's easy enough to reverse-engineer a design, and there are entire companies whose existence is solely to provide such services.
Intel had schematics for all their motherboards. Unfortunately they've left the motherboard business.
Interesting, I didn't know that. Was that true all the way up until they left the business? What was the most recent generation of motherboards for which they published the schematics?
Gets pretty hard with multilayer boards and smd components
To be honest, I think layouts like this are done for practical reasons like space constraints rather than to annoy reverse engineers. If you are serious and can disassemble a board, you will find these components immediately.
https://developer.puri.sm/Librem5/Hardware_Reference/Chestnu...