Vulnerable transistors threaten to upend Europa Clipper mission | Hacker News Reader

Vulnerable transistors threaten to upend Europa Clipper mission

science.org

177 pointscratermoon105 comments

They found out about the failing transistors via colleagues at a conference. Have any of you learned of something of this magnitude in the same way? It got me thinking that I need to interact with my fellow devs more often.

This is much more common than one would think.

This is a classic thing with Industry, they qualify a process that is working and shows good performance, but this process needs to be changed for reason XYZ, often because it is maybe a bit too expensive or doesn't align with the rest of their processes. The small change in the process wasn't that small and takes a little while to be identified because by the time you catch it you might be further down the line and this would be caught by a QA process and not a QC process, that might have deemed at that point not necessary because you had no reason to fault the part.

The second part is that some things are rated and verified but not tested extensively, since you might have prototype you might misdiagnose a failure of a component for a behaviour of your prototype, when in fact you had a deeper problem, but timelines with the added fact that so far you didn't think about that problem because it shouldn't have been a problem can catch you really off guard. This is usually where people testing the same thing in an exotic environment can ring alarm bells for others and that often happens at conferences...

People often under estimate how much you can get bitten in the back by such little details that become huge details.

Depending on the electronics and where the MOSFETS are, I would be them I would probably trash the electronics, take the spare that they had, validate components that get in and rebuild a control box and re-integrate it, provided that this is doable. It's expensive but provided that you have no choice that gives you a backup system that you can test code on before pushing it on the actual probe and might help for problem solving by being able to do measurements and test on the actual setup... Provided that they have the time and resources. Otherwise I wouldn't YOLO it given the fact that it might just straight up not work at the moment you need it the most and a little delay is better than nothing and they can spend the time re-checking part of the design that might also be weaker...

But heh, who am I but a random guy on the internet...

Another example along these lines: scurvy! [1]

They found it was cured by lemon juice, but they didn't understand the details. Over years, they switched to lime juice (less vitamin C), put it in copper pipes (leaches vitamin C). But ships were faster so there was more fresh food available, masking the problem. Then scurvy starts mysteriously popping up again 100 years after it was first "cured."

Hard to keep track of the effects of all the details in the face of various co-dependent things changing simultaneously. Recipe for surprises.

[1] https://www.bluesci.co.uk/posts/forgotten-knowledge

Yep. Chatting with other practitioners is a powerful way to learn how things actually work. There are tons of things that "everyone" knows that are not well documented, and therefore unavailable to people outside the network.

This is a more-consequential example of the things you can learn by chatting with others; it is an extreme example of, "Hey, are you guys using components from Widget Inc.? Their datasheets are good, but sometimes we get a bad batch."

Those little things can save you a ton of time. In this case, it may have prevented mission-failure.

Part of the blame falls to NASA, too. If the outcome is your responsibility, then open-loop trust of a vendor for a known failure-mode may not be acceptable. Integration rad-hard testing may be requisite.

In the spacecraft environment, qualifying components is very difficult -- there's a good chance that NASA has these MOSFETs on an approved list because they've worked well before and have had few (or known) faults. They're probably not on that list anymore.

This is what forums like this one are for. Ordinary news isn't going to have more than a passing mention of the xz hack, or log4j, or meltdown, or heartbleed. Find (or start) a private group chat for technologists you know to share news like this.

I can't believe the manufacturer didn't alert them and they had to hear it from another customer. Surely the manufacturer wouldn't want to be named as the reason that a spacecraft orbiting Jupiter went dark due to their faulty components.

The article mentions that the defense sector discovered the issue. Rad hard defense electronics have more stringent TID (total ionizing dose) requirement than space, due to a need to survive in nuclear war scenarios. Space usually caps out at 100 krad, with some very stringent environments needing up to 300 krad. Defense can go all the way up to 1 MRad in some cases.

My guess is the parts failed TID at the more stringent levels, and Infineon didn't follow up with NASA or their contractor because they assumed that NASA was okay with the lower rad tolerance levels typical of space. Usually that would be the case, but Europa Clipper is special because it's going to an extremely harsh radiation environment.

The big question for me is: did the Europa Clipper program order a lower TID and try to upscreen, or did they order the high TID part? If it's the former, it's on NASA. If it's the latter, that's extremely concerning because Infineon should know that nobody orders expensive high TID parts for funsies, and they should have followed up with all customers as soon as they confirmed there was an issue. Just assuming NASA over-specified a part is absurd. The rad hard electronics market is small, everyone knows each other. Trust is king.

Finally, I'm not sure if it's the part in question, but it looks like Infineon discontinued their 1 MRad MOSFETs in 2020, citing low order volumes: https://irf.com/product-info/hi-rel/alerts/fv5-d-21-0004.pdf. In the light of this reporting, I have to wonder if there was more to it than that?

> and Infineon didn't follow up with NASA or their contractor because they assumed that NASA was okay with the lower rad tolerance levels typical of space

It's more likely that Infineon's folks talking to NASA were equally clueless about this change.

Ultimately, NASA bought a part with a specified TID tolerance. Any manufacturer of space qualified parts keeps detailed records of lot acceptance testing as well as who purchased from that lot. The reps interfacing with NASA didn't necessarily need to know that there was a process change, but as soon as test failures below the datasheet spec were communicated from customers and confirmed, Infineon's quality department should have immediately reached out to NASA (or more specifically NASA's contractor working on the electronics).

" Infineon's quality department should have immediately reached out to NASA (or more specifically NASA's contractor working on the electronics)."

Is there any actual evidence they didn't reach out to every single buyer of the electronics?

The article goes out of its way to say Infineon did not contact NASA. But even in your description, they would not have, they would have contacted NASA's contractor working on the electronics.

I still go back to "if there was actual evidence that Infineon did not notify who it was supposed to, the article probably would have cited it". There isn't, so they instead cast aspersions.

Instead they make a bunch of hay about a statement from Infineon that seems totally innocuous - they didn't notify people they didn't know about. Shocker.

Look, i actually hate Infineon - i've been forced to try to make their wifi and bluetooth modules work properly before ;-)

But this kind of lazy-at-best journalism doesn't help anyone.

Or maybe those making a living on selling a product claiming certain parameters should raise their voice when those parameters are not met, regardless if that product is used for space travel or turn on and off a ketle for f's sake.

This is the "status meeting paradox."

Personally, I despise status meetings. 99% of them are worthless fluff.

But, every now and then, you get something like this.

I think that highly usable dashboards are a good way to deal with this.

It's possible that AI could be a big help, here.

Also the hallway conversation thing. Most of the time it’s small talk and minor social interaction, every now and then it’s critical out of band information that would not have shown up in normal processes.

To me it's a matter of fostering serendipity. and a bit ironic that research has shown conferences to be a great place for serendipity to take place, as that's what happened here.

I experienced this kind of situation, where only by chance conversation was a crisis averted, very much at my last FT. So much that I'm working on a startup for fostering serendipitous communication for remote teams, like private notes from coworkers left on stackoverflow questions (or anything on the web)

Perusing Slack has become this for me

Probably inevitable these days given hallway conversations are going to be a pretty random thing. Of course, assumes someone needs to think something is important enough to put in chat and doesn't mind putting it out in public. (Ignore $XYZ project that other group is doing. It's got all sort of problems.)

Yeah, the ephemeral, no-record nature of verbal hallway conversations makes them more candid.

Your highly usable dashboard will get filled with 99% of worthless fluff just because it's there and somebody feels the need to always say something.

Have you even been in one of those meetings that just won't finish despite everything being done? Making it written doesn't solve the problem. Instead, it makes it worse.

On point. This is also why good CI/CD automatically alerts users of major issues. It's just not a thing humans are good at to pay attention to a long stream of mostly boring information.

Computers are good at this though.

Now the only question is how you can automate the spec comparison such that issues with the spec and the parts used can be automatically compared.

And that starts with a computer readable spec that is updated by the manufacturer.

That worthless fluff makes the dashboard look impressive when you put it up on large NOC type screens when you walk people through on tours though

"We are at the point where everything that could be said has been said, but not everyone has said it."

Good point.

AI it is, then...

Yes, all the time. This is normal. Big news of problems travel through back channels. Nobody is gonna announce their big fuckup for the world to see, unless compelled by law, and even then most won't do it. We had to sign NDAs to find out about severe Intel processor silicon bugs. Obviously you're not gonna read that info on Hackernews afterwards.

> The transistors cannot simply be replaced. Clipper’s aluminum-zinc electronics vault, meant to provide a measure of radiation resistance, was sealed in October 2023.

Surely they can be replaced! Humans (in cleanrooms) can put their hands inches away from them. Here's a picture, admittedly from 2022, before it was sealed:

https://europa.nasa.gov/resources/342/electronics-vault/

It's not the same as saying they've failed and are in orbit around Jupiter and so can't be replaced. They simply need to open and re-seal the vault or build and seal a new electronics vault before October.

Here's a photo of the vault as it was about to be sealed in October last year: https://www.nasa.gov/missions/europa-clipper/how-nasa-is-pro...

In the meantime NASA built a whole spacecraft around it and the craft went through numerous tests. If they'd have to undo and redo all of that, they may even miss the 2025 backup launch date. I'm sure NASA is very eager to seek other ways to deal with this problem.

Ok, that sounds a bit more convincing... I'm not the OP, but I was just trying to imagine why a "sealed vault" would work better against cosmic radiation than a more prosaic screwed shut container.

On one hand, it must be baller to work on assembling spacecraft. On the other, wearing those gowns all day must be terrible.

Do you understand that testing and certifying everything takes months and is a significant portion of the cost of the mission? And plenty of these components and closures were designed for single use, so they would have to be replaced. This is not a desktop PC.

>[Infineon] has already corrected the mistake, but Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Fitzpatrick said. “They did not realize it was going to affect us.” Infineon did not respond to a request for comment.

Not exactly responsible disclosure! NASA buys rad-hard transistors, and Infineon "didn't know what they'd be used for"?

I bet NASA buys rad-hardened electronics by a truckload, and buys from distributors, not Infineon directly.

But it's a reasonable idea to notify all potential large consumers that are likely to have bought your specialty product; these are not numerous, and the impact may be large (as in this case).

Rad hard parts are always sold direct from manufacturer in my experience - adding a distributor just muddies traceability, which is critical in space programs. There's usually a lot of communication between the quality departments of the manufacturer and the buyer, as test reports need to be transferred to the buyer for their records. Infineon almost certainly a list of everyone who has purchased these parts as well as the phone number for their quality control department.

They really don't. Each individual project is sourcing their parts on their own, and even when there's a subcontractor involved we're often talking to the manufacturer as well. So Infineon almost certainly has some record that these parts were specifically for Europa Clipper.

That's certainly possible, but the distributer should have notified NASA. Maybe that email is sitting in someone's inbox.

Distributors will notify you of recalls, but no distributor of electronics i've worked with notifies you of erratum (and it would be really annoying if they did, honestly :P)

This reads like it should have been a recall. But that's hard to tell unless I knew exactly the specifics of the issue.

Did NASA assume these were rated higher then they were? Did Infineon make a mistake in documentation, or did they straight up not test them or test them incorrectly.

There's no "consumer rights" in B2B transactions.

Unless contractually specified otherwise, it's generally up to the buyer to check the delivered goods for defects and report those without undue delay*. If this is not done, the goods are deemed to have been accepted.

Sure you can contractually specify that the product has to meet certain specs and pay extra for the seller performing QA, but the default often is "you're buying whatever comes out of our factory, check the goods yourself on delivery". The reason things are done this way in the business world is that it is generally cheaper to accept certain failure rates than to perform testing at every step of the supply chain and add a whole lot of bureaucracy and complications because of returns.

Whether custom contracts existed in this case is unknown, but it is likely that Infineon notifying customers was already a courtesy. They could've just said nothing.

* Under German law, which likely applies here since that's where Infineon sells from.

IANAL, but I would think Infineon's data sheet and quote would constitute the "offer," and NASA's purchase order the "acceptance." IIANM, this meets the minimum requirement to establish a "contract" (usually called an "agreement" these days).

If the MOSFETs don't meet the specs on Infineon's data sheet, including rad hardness, then Infineon would be in breach of contract.

Is my reasoning correct?

If NASA accepts the delivery of those things and doesn't check for & report defects*, then outside of willful deception on the Infineon's part, it's not the Infineon's problem anymore. It is the responsibility of the buyer to check that the items are as specified. If the buyer neglects that responsibility and signs for the delivery, the seller is off the hook.

German law differentiates between "open deficiencies" and "hidden deficiencies". If you neglected to properly check for an open one, that's on you. You now have no warranty under the law. In case of a hidden one, which will likely only show during large-scale production and can't really be detected beforehand, you have to immediately report it once you discover it, and it is your responsibility to document & prove that you did so without delay.

Under this system it's up to the buyer to decide how much reliability they need. They can forego testing and save money because it's not important to test every single screw when building a garden shed, or they can rigorously test every single thing because they're building a spacecraft.

* It is enough to prove that you did perform checks. If you got unlucky and the random samples just happened to be good, you are still protected. But if you didn't check at all or not sufficiently, you're screwed.

What an embarrassing moment for Infineon. IME their products tend to be very nicely engineered... and onerously documented, but that's probably a good thing if you're NASA. This, though, is concerning. With companies like Infineon, Analog, ST, etc. you're literally buying black boxes and an unenforceable promise that those black boxes will behave the way the datasheet says. This is a pretty egregious breach of trust, and Infineone really must do better to uphold their image.

Does this seem like a one-off mistake or is this a systemic problem that is likely to strike again?

Hard to say. I sure hope it's a one-off blunder.

It also implies that unless you using their products on a high-profile space mission Infineon doesn't plan to notify customers of known product defects. I'm not sure how Infineon thinks "if only we had known that we would have told you" is going to go over well

What was the change in the composition of the transistors?

I don't really know this field, but might they have switched away from silicon-on-sapphire?

>> did not know what the transistors would be used for

There are so many types of radiation that I do not think it unreasonable that they only notified customers who used these devices in particular environments. Most military use would be near radio transmitters (radars) or nuclear reactors (navy). Neither use case are an exact match for the radiation environment of Jupiter orbit.

I don't know if that's actually true, but in this case the article specifically calls out classified satellites, so in this case the original problem was also with space-based radiation.

Did Infineon not issue a GIDEP Notice for these parts?

If not, does that mean that maybe NASA is using them outside of their designed spec?

Europa Clipper also used a new approach for designing spacecraft. It's NASA's first major spacecraft designed with Model Based Systems Engineering(MBSE)[0]. Using diagrams in SysML to keep track of power use and interfaces is supposedly better than using spreadsheets

[0]https://ses.gsfc.nasa.gov/ses_data_2021/210728_Bayer.pdf

Oh, I used to work on this :)

For keeping track of power use and interfaces specifically it turns out doing it all with SysML diagrams wasn't so great. Aside from all the pointless futzing around with boxes and arrows the model eventually became so huge the authoring software could barely handle just opening it up. So it must have been shortly after these slides when all the power use tracking was shifted to a custom tool with a more tabular user interface that we were already using for tracking electrical interfaces (slide 15) with version control in git.

Unrelated question: how did you manage tabular data in git? It’s always a struggle to diff and merge changes.

Yeah, like 'baq said the data wasn't stored in a tabular form, it was actually XML. So sometimes you could just look at the textual diff and it would make perfect sense, although it wasn't expected users would work with XML at the source level.

There was also a semantic object-level diff we got for "free" by virtue of building on top of the Eclipse Modeling Framework. It was integrated into the Eclipse git UI and could help resolve merge conflicts without having to touch the XML directly, but merge conflicts were still annoying to deal with so generally engineers coordinated with each other to not touch the same part of the model at the same time.

Normally for review though I think users tended to compare reports generated from the model rather than trying to diff the source model files directly. There was a sort of automated build process that took care of that once you pushed your branch to Github.

not OP but the usual applies - data is not actually stored as a table in git, tables are an UI thing. git would store standard issue json, xml or whatever custom git-friendly format is used by the tool.

> Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for,

People are reading this as Infineon didn't know that the parts were going into a probe when it's far more likely they meant they didn't know how the transistors are being used in that probe, which might have a large effect on whether or not the problem will affect them.

Is more shielding not the obvious answer? A thin sheet of lead around the sensitive parts should do the trick.

(Note: I'm not a physicist and have no idea what I'm talking about in this domain)

In space, a thin sheet of lead is not radiation shielding but a radiation amplifier.

The problem being that high-energy cosmic rays are unlikely to interact with the lightly built spacecraft, going right through it. But if you add a thin layer of a good radiation shielding material, then there is substantially increased chance that they will interact with that material, and produce a very large spray of secondary particles. And those secondary particles will also be going fast enough that when they hit more shielding material, they will also result in more particles.

Then some of those secondary particles will be neutrons, which will easily penetrate the thin shielding (lead half thickness for 4MeV neutrons is 68mm), and irradiate the surroundings.

This has been very clearly demonstrated on the ISS, any metal tool has substantially higher radiation levels around it.

Thank you for this post. I was wondering if a thin lead sheet would be beneficial for the cockpit ceiling and maybe aisles of jetliners to protect the crew from the prolonged exposure to increased radiation. Do you think this is a bad idea for the same reasons as the spacecraft? (Of course there are other materials besides lead, that was what first came to mind because I incorrectly thought it was a panacea for all radiation types).

Air pressure at airliner altitudes is still about 20-30% of the sea level value. That means 20-30% of the atmosphere is above that—a column of mass equal to 2-3 meters of liquid water.

A thin lead sheet would be a rounding error next to that.

This is an oversimplification that's rather wrong, but: a decrease in altitude of just 300 meters, at airliner levels, puts an additional atmospheric mass equal to ~1 cm of lead (Pb) above your head.

Have you seen the explanations of radiation where they say flying (as a passenger) is about equal to the dosage of a dental X-ray (or something similar)? Someone who spends their career getting exposed at that rate might be worth making them a shield.

It's *not possible* to make a shield. The cosmic radiation that reaches that altitude is highly energetic and highly penetrating—enough so to go through 3 meters of water—and would be completely unaffected by lead sheets. Any easily-shielded types of cosmic radiation have already been blocked by the atmosphere.

It's not worth it. Like, you could mandate planes fly 100 meters lower, but a lead coating would be so heavy. Let's say a 737 with a max weight of 70 tons. Covering the top half with lead would mean a 200 square meter sheet, and at 1cm thick it would weigh more than 20 tons.

I had read elsewhere that water is a useful shield. A quick search found this document, that mentions the danger of secondary particles.

https://www.nasa.gov/wp-content/uploads/2009/07/284275main_r...

The rad vault on Clipper is an aluminum-zinc alloy, not lead. There are different kinds of radiation to worry about (alpha, beta, gamma, neutron, protons, heavy ions), and I think certain shielding approaches good for one aren't always good for the others.

Different sources of radiation interact with electrons or nuclei (1:1 with number of atoms) or nucleons (individual protons/neutrons, 1:1 with the mass). For instance, neutrons bounce off nuclei in nuclear reactors, and the lighter they are, the more energy the bounce can siphon off from the neutron. So having more, lighter (low-Z) nuclei (hydrogen in water and carbon in graphite are commonly used) provides better slowing of the neutrons vs. heavier (high-Z) elements, like lead.

Smashing ions (alpha, protons, heavy ions) into materials can also cause a https://en.wikipedia.org/wiki/Particle_shower

> Is more shielding not the obvious answer? A thin sheet of lead around the sensitive parts should do the trick

Lead "is effective at stopping gamma rays and x-rays" [1]. Jupiter's radiation comes from "trapped particles [that] are about ten times more energetic than the ones from the equivalent radiation belts of Earth" and "several orders of magnitude more abundant" [2]. When those encounter lead they cause bremsstrahlung radiation [3], a sort of subatomic shrapnel that can be more dangerous than the original radiation.

Lead is also heavy, which means not only increasing the mass of the spacecraft, but its balance and thus propulsion profile. That might mean upgrading and moving thrusters and propellant tanks--in effect, a complete redesign.

(It's a good question that doesn't deserve to be downvoted.)

[1] https://en.wikipedia.org/wiki/Lead_shielding

[2] https://www.spenvis.oma.be/help/background/planetary/traprad...

[3] https://en.wikipedia.org/wiki/Bremsstrahlung

Europa Clipper electronics are contained in a 9.2 mm thick aluminum-zinc vault.

https://europa.nasa.gov/resources/342/electronics-vault/

> Europa Clipper electronics are contained in a 9.2 mm thick aluminum-zinc vault

Were it designed today we'd probably dope it with titanium [1][2].

[1] https://www.tandfonline.com/doi/full/10.1080/10420150.2023.2...

[2] https://www.sciencedirect.com/science/article/abs/pii/S01491...

Could they find some margin to make it a bit thicker? I know this would increase the weight but if my image of how big this electronics vault must be I'd imagine they could find something less critical to shave off to offset it.

Unless the launch is postponed 2 years, I think any redesign of the vault at this point is unlikely. Clipper was originally designed to be launched on an SLS rocket and that was swapped out for a less powerful Falcon Heavy* so there isn’t going to be much room for extra mass. Additional mass may require more planetary "slingshots" and add more years before Jupiter arrival.

Hopefully SpaceX is able to resolve its Falcon second stage problems before Clipper is scheduled to launch.

* There were some discussions about adding a Thiokol Star 37 or Star 48 apogee kick motor to the Falcon Heavy stack for Clipper but for various reasons this didn’t happen.

https://en.m.wikipedia.org/wiki/Star_(rocket_stage)

Went searching for the "various reasons". Found this:

> Falcon Heavy rocket, having three launches under its belt, has proven more powerful than originally anticipated. Previously, it was thought that launching Europa Clipper on a Falcon Heavy would require a “kick” stage — essentially a small booster attached to the top of the rocket. The Falcon Heavy’s impressive performance has made that unnecessary. Moreover, mission designers at Jet Propulsion Laboratory have found a path to Jupiter called a MEGA trajectory: after launch on a Falcon Heavy, Europa Clipper would fly to Mars for a gravity assist, and then return to Earth for another, and then on to the Jovian system. (The mission previously believed that the rocket would necessitate a Venus gravity assist, which would require special thermal protection for the spacecraft.)

> The window for a MEGA launch opens in 2024 and would take only three years longer than an SLS flight. A Falcon Heavy expendable launch is about $150 million. A single SLS launch is now estimated to cost $2 billion.

Source: https://www.supercluster.com/editorial/europa-clipper-inches...

> I'd imagine they could find something less critical to shave off to offset it

You’re still changing the spacecraft’s balance. Imagine moving one of an airliner’s engines a foot to the left. It can be done. But it’s a big change.

Now consider that “modern jet airliners have…useful load fractions, on the order of 45–55%,” while orbital rockets’ payload fractions are “between 1% and 5%” [1]. Deep space craft are another order of magnitude more sensitive.

Adding a little shielding here and there is the aeronautical equivalent of hanging a bag of bar bells off the tips of one of the wings.

[1] https://en.m.wikipedia.org/wiki/Payload_fraction Note: useful load != payload fraction, but within orders of magnitudes they’re comparable

If you are going to the trouble to take apart and redesign the system, it would be far easier and less dramatic to just replace the possibly out of spec transistors.

Yes, and if they had larger mass budgets they could over-engineer things like shield thickness to have wider safety margins, and mitigate unexpected problems like this one. One can speculate future space probes generally will become more more reliable, as the the cost of mass-to-orbit goes down, and engineering constraints become looser.

(I wonder if Starship is useful for this type of problem: if you could adapt the orbital-refueling method to serve as radiation shielding, and put an electronics vault in the middle of the propellant tank? Could you adapt Starship into a spacecraft bus in this way?)

Water tanks are the most likely source of radiation shielding: propellant tanks get used up and go empty, while for any lengthy mission, water is either going to be recycled back into the tanks or you will have to take blue water tanks and over time turn them into grey water tanks, either way you will have those tanks much more filled than the propellant.

These spacecraft always have tons of instruments for measurings along loads of different axes. For example, a magnetometer specifically designed for testing a hypothesis about Europa's magnetosphere. Looking at the wikipedia page it seems like there are about a dozen of these. Perhaps worst case scenario they could determine which one was least critical relative to its weight and eliminate it to increase the mass budget.

1. You’d need more than a thin sheet of lead. The radiation in space can be very energetic. It can easily penetrate several cm of shielding and if it is absorbed, you get secondary radiation.

2. Even a thin sheet of lead may be too heavy.

My favourite bit about this project:

> In June 2022, project scientist Robert Pappalardo revealed that mission planners for Europa Clipper were considering disposing of the probe by crashing it into the surface of Ganymede for Europan protection purposes, in case an extended mission was not approved early. He noted that an impact would help the ESA's JUICE mission collect more information about Ganymede's surface chemistry

What about Ganymedian protection, eh? The Ganymedians, should they exist, are going to be furious.

Also it is weird that the outcome of the EJSM divorce (originally there was going to be a joint NASA-ESA mission to the Jupiter icy moons; Europa Clipper and JUICE are a result of the breakup), is that America explores Europa and Europe explores Ganymede, as the other way around would be less confusing.

If the Ganymedians got angry and attacked us, that would be a great scientific advancement.

> Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for,

Either the parts were in spec or they weren't. Which is it?

That's not how specs work.

When the requirements for a part are specified, it is based on assumptions that may or may not hold true.

For example, if an issue tends to be all or nothing, then testing a small percentage of a lot should reasonably be expected to catch an issue. So you might specify that 1% of these transistors be tested and so long as that 1% passes the rest are considered good. If let's say there's a process change and lots become more variable, the confidence with which you can say the others are good based on that 1% testing goes down, but you are still testing to the same standard that you were before, which is what the specification calls for.

The issue gets even more thorny when issues are conditional. For example a part might meet the voltage specification, the temperature specification, and the radiation specification individually, but when you put that same part simultaneously in a high voltage, low temperature, and high radiation environment it doesn't perform as well. Or perhaps one component used downstream of a particular other component has an effect. Perhaps the most basic example is oversized but in tolerance shaft meets undersized but in tolerance hole.

>For example a part might meet the voltage specification, the temperature specification, and the radiation specification individually, but when you put that same part simultaneously in a high voltage, low temperature, and high radiation environment it doesn't perform as well.

I am not disagreeing, but at some point humanity should switch to actual probabilistic device models instead of vague datasheets. Imagine every datasheet has a .sample() function and you get a randomized SPICE model as if it came from the manufacturing line, you can draw a 100 and plot the properties. Want to measure the dynamic range of some ADC? A-weighted or not? Instead of specifying values highlighting figures of merit, each such figure of merit corresponds with an explicit SPICE circuit that measures that figure of merit on one and the same generator for random SPICE models with that specific part designation.

If a brand tries to fool its customers by insinuating a desirably high or low value by changing the test method, its immediately clear. A user may specify his own test circuit, or reuse the test circuit from the interactive datasheet from a competitor etc in order to make apples to apples comparisons for different DUT's.

So can we say the transistors of Clipper are ... clipping?

More seriously, I'd be interested to learn what failed in the quality assurance process, as NASA & its suppliers have a legendary reputation in these topics. RCA will be enlightening.

From the article: "Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Fitzpatrick said."

They might not have "known", but come on, you're selling radiation-hardened chips to NASA. You can sure make an educated guess that they might be used for a probe.

I'm guessing there's a clause missing in the contract that says Infineon must disclose all known problems to NASA regardless of how the chips will be used.

Regardless, there are some people at NASA to whom 'Infineon' is now a curse word.

"I'm guessing there's a clause missing in the contract that says Infineon must disclose all known problems to NASA regardless of how the chips will be used."

The article doesn't say or even imply that NASA has any contract with Infineon. It seems much more likely they are buying the chips through one of their approved distributors.

Without something saying that NASA bought directly from infineon:

1. It's not obvious how they would know who they sold to.

2. It's not obvious how they could get the information out beyond how they usually do it - issuing erratum notices.

Honestly, it feels like the article goes out of its way to try to imply Infineon should have notified NASA, but gives no data to suggest it had any idea at all what was going on.

If they had data that infineon and NASA had a contract, they would have put it in the article and used much stronger language. All these contracts would be public and are easy to find.

The fact that they don't have anything in the article about this suggests the contracts don't exist, and as usual, they are just using implication instead.

Rad hard parts are basically never sold through distributors. Strict lot traceability is a requirement on space programs (to avoid the issue discussed in the article). The quality departments at the manufacturer and buyer also need to communicate a whole bunch of stuff (requirements, test reports, etc) which defeats the purpose of the insulating layer of a distributor. Also, while these parts are expensive (my rule of thumb is to add 2-3 zeros to the cost of a commercial part to estimate the cost of a rad hard version), they are low volume, so there's not a whole lot in it for a distributor. The contractor working on the electronics almost certainly purchased these parts directly from Infineon, and Infineon would have had records of who purchased parts from which lot.

I'll assume everything you say is right :)

The question here is whether Infineon had a contract with NASA or otherwise should have known these were sold to NASA.

Again there is nothing cited in the article that says "yes".

If you've got data that says yes, awesome, what is it?

The fact that they found out about this accidentally at a conference is, all by itself, extremely strong evidence that Infineon didn't notify whoever they should have for the Europa Clipper mission, whether that was NASA itself, an in-house contractor or an external subcontractor.

Other articles[1][2] mention that the transistors came from International Rectifier which was bought by Infineon ten years ago. Maybe Infineon wasn't aware because NASA acquired the transistors through their IR subsidiary. IR provided transistors for the JWST and even for Hubble[3], so they probably were NASAs go-to supplier for this kind of hardware.

[1] https://www.eenewseurope.com/en/nasa-tests-infineon-power-mo...

[2] https://hardwarebee.com/electronic-breaking-news/nasa-tests-...

[3] https://www.eenewseurope.com/en/ir-hirel-rad-hard-components...

"They might not have "known", but come on, you're selling radiation-hardened chips to NASA. "

But do people ever actually "invoice NASA" for components. It was probably one of 100 different sub contractors building the actual circuits to NASA specifications, i.e. it was lower in the chain rather than NASA itself.

(Doesnt excuse the non-disclosure to those subcontractors)

>But do people ever actually "invoice NASA" for components

Yes, absolutely they do. I'm not a part of this mission, but I'm currently working on another NASA spacecraft mission. I don't know the percentages off hand, but a substantial portion of our spacecraft is built in house with parts purchased directly by NASA from the manufacturer.

Regardless, there are lines of communication to subcontractors. The mere fact that they found out about this at a conference is significant evidence that Infineon didn't notify who they should have.

Off-topic, but when components are sourced directly from the manufacturer do you have to buy in bulk? I figured you didn't just go on Mouser or DigiKey, but I would think manufacturers don't like dealing in small amounts.

For spacecraft parts, they absolutely don't mind (they're charging for the privilege of course). For the parts I'm familiar with, we generally buy both the necessary flight-rated components (both enough to build the vehicle and some number of spares) and a number of unrated components used in various test apparatuses in a single order. Once you get down to the level of stuff that's not even a flight-test fixture, we can indeed source parts from pretty much wherever. The biggest issue then become US government procurement rules that require us to buy American, but I'm pretty sure I've seen at least Mouser get used before.

Well, I'm glad they didn't find out, by the board crapping out, around Jupiter.

> The transistors cannot simply be replaced. Clipper’s aluminum-zinc electronics vault, meant to provide a measure of radiation resistance, was sealed in October 2023. Barring an indication that the faulty MOSFETs will cause catastrophic failure, the agency will likely seek to continue with the launch—although backup windows are available the next 2 years.

Yeah, but if they figure out that the transistors are problematic, at least, they can do something about it, even if it does delay the launch.

They may also use this as a spur to wargame failure mitigation strategies, so they'll be ready, if they do go belly-up.

This happened already at least once with a couple Soviet Mars probes - they found a batch of transistors they used were faulty and deteriorate much too quickly in space - after the probes vere already launched.

IIRC some did not even reach Mars while others failed soon after orbit insertion due to various sub-systems failing.

Right. Like for all the issues here, it's still better to find this out now.

Sounds like Infineon may owe someone a new satellite soon. At least if it can be shown that they sent NASA bad parts and didn't notify them in time to prevent this failure.

Id like to know how this turns out - ie. hiw they decide to mitigate. Where's the easiest / best place to check in a couple months for a followup?

Germans and space industry, what could go wrong

the Mosfets are actually from IR HiRel which is an American legacy space company

I am eagerly awaiting probe manufacturers learning anything at all from Ingenuity.

The Ingenuity is a really interesting project with some important lessons for spaceflight with unhardened CotS parts. However, I would argue that it and the Europa Clipper are two very different designs for two very different environments.

While mars is an elevated radiation environment when compared with earth, the Jovian radiation belts are on a whole other level, particles up to 1-2000 MeV are fairly common. To put that into context, a medical radiation beam therapy deals with 2-300 MeV on the absolute highest end. To get into the 1-2000 MeV range you generally are talking about energies found in the low end of particle accelerators. Ingenuity mostly had to worry about Total Lifetime Dose (TLD), one example of a TLD issue is dopant migration induced by high-energy heavy ion collisions which can change the on voltage of a transistor. At high energies you can have single events with enough energy to cause fatal latch-ups. For instance modern rad-hard FPGAs start encountering major issues around 60-70 MeV.

Furthermore, these parts are power MOSFETs which control power for whole subsystems so their reliability is critical to the operation of the spacecraft. In addition, the biggest issue here is not just that there were issues that were addressed and fixed, it's that Infineon didn't issue an errata to the datasheet or inform NASA of the issue. As a result there are now transistors littered throughout the spacecraft which don't meet the radiation needs. This is going to require reworking the boards, re-validation of the subsystem, and re-integration of the subsystem into the spacecraft. This all comes at a non-trivial impact to budget and timelines which is to say nothing about what this does to the launch window the project was trying to hit for gravity assist / proximity.

I hope you find this informative! :)

EDITS: Spelling and an "is"

Are you referring to off the shelf components used for the helicopter?

https://en.wikipedia.org/wiki/Ingenuity_%28helicopter%29?wpr... (in case anyone wanted to refresh their memory)

DAE when i read a sentence like this

> Engineers at NASA’s Jet Propulsion Laboratory (JPL), which leads the development of Clipper, discovered the problem in May after talking with colleagues about a classified satellite at a conference.

...I immediately think "that's what they want us to think"

if classification of projects means anything, the true intent of the passing of dis information is something dat we can only guess at.

fnord.

1. Sell bunch of radiation-hardened parts to NASA. 2. Find out the parts you sold to NASA don't meet the specs. 3. Don't tell NASA, because NASA didn't tell you what those parts would be used for.

This is criminally incompetent on the part of Infineon. WTF, NASA could use those transistors for a fancy inteliggent toilet FWIW, it doesn't matter, NASA doesn't have to tell you how they are going to use those parts. They bought parts based on a fucking SPECIFICATION, and if the parts you sold them don't meet the specs, you communicate immediatelly with the customer offering a replacement for free.

Really, someone should be jailed for that.

Jailing people for mistakes is a good way to ensure that mistakes are kept hidden.

There are mistakes, and there is negligence.

Reply on news.ycombinator.com