Hacker News Reader

Comment by opensores | Hacker News Reader

(some rambling questions here that are only semi-related:)

NASA is really anal about radiation concerns, insisting the caches have ECC (or they run it with cache disabled). For the instruction cache this makes sense (as well as any higher caches that would store instructions).

With modern CPU's having so many cores and threads being so cheap, couldn't NASA use these for redundancy?

My second question, how much more resilient are older processes with larger feature size compared to modern 6/7nm ? Compare modern with older (14nm/22nm finfet, 28nm/32 soi) and ancient (65nm or larger).

View on HN

> With modern CPU's having so many cores and threads being so cheap, couldn't NASA use these for redundancy?

Power constraints and heat dissipation constraints.

https://arstechnica.com/science/2019/11/space-grade-cpus-how...

> BAE RAD5545 is probably the most powerful radiation-hardened processor available today. Fabricated in the 45nm process, it is a 64-bit quad-core machine clocked at 466MHz with power dissipation of up to 20 Watts—and 20 Watts is a lot. A Quad Core i5 sitting in a 13-inch MacBook Pro 2018 is a 28 Watt processor. It can heat its thin aluminum chassis to really high temperatures up to a point where it becomes an issue for some users. Under more computationally intensive workloads, fans immediately kick in to cool the whole thing down. The only issue is that, in space, fans would do absolutely nothing, because there is no air they could blow onto a hot chip. The only possible way to get heat out of a spacecraft is through radiation, and that takes time. Sure, heat pipes are there to take excessive heat away from the processor, but this heat has to eventually go somewhere. Moreover, some missions have tight energy budgets, and they simply can’t use powerful processors like RAD5545 under such restrictions. That’s why the European GR740 has power dissipation at only 1.5 Watts. It’s not the fastest of the lot, but it is the most efficient. It simply gives you the most computational bang per Watt. The HPSC with 10 Watt power dissipation comes in at a close second, but not always.

45nm is massive for a high-performance processor. Rad-hardening compute is frankly overrated. Have multiple and compare state at a high-rate.

But you can't power multiple on the energy budget that you've got.

Perseverance is running on 110 watts ( https://mars.nasa.gov/mars2020/spacecraft/rover/electrical-p... ). The RAD5545 uses 20 watts of that budget. Tripling it means it doesn't have enough energy to power the rest of the rover.

The Mars MAVEN ( https://spaceflight101.com/maven/spacecraft-information/ ) is using a RAD-750 which uses 10 watts of power ( https://en.wikipedia.org/wiki/RAD750 ) (also used in the JWST).

The RAD-750...

> The processor can endure radiation doses that are a million times more extreme than what is considered fatal to humans.

And while you may argue "yes, that seems a bit excessive" - the goal isn't "it can run in those extremes" but rather...

> Also, RAD750 will not suffer more than one event requiring interventions from Earth over a 15-year period.

> “The RAD750 card is designed to accommodate all those single event effects and survive them. The ultimate goal is one upset is allowed in 15 years. An upset means an intervention from Earth — one ‘blue screen of death’ in 15 years. We typically have contracts that (specify) that,” said Vic Scuderi BAE Business Manager.

... because it can't be replaced easily.

From Wikipedia:

> The CPU can withstand an absorbed radiation dose of 2,000 to 10,000 grays

You can have one disruptive rad event every 10 minutes if you compare and correct state between three or five computers at something 100 Hz.

And a more recent computer that isn't designed for rad-environments will automatically be way more efficnent than a rad-hardened one.

What set of processors would your recommend putting in Perseverance with a 110 watt power budget? Or on JWST with its heat dissipation requirements?

Yes, but you have to compare the complete state of each computer for this to work, including every bit of memory & cache. You can't have much RAM if you're planning to compare them in a few milliseconds.

If you try to do this without comparing the whole state, and only comparing the outputs of some control algorithm between the computers, one of them could have some internal state corrupted and remain unnoticed for a long time, long enough for other corruptions to happen on the other computers, and then you have no straightforward way to figure out which one to trust and which to reset.

I do not buy the power argument. A modern chip is orders of magnitude more power efficient, especially at low voltage. You could easily have multiples of these within the same power budget.

But smaller size increases the chance for event (which might be compensated) but also increases chance for fatal damage to happen.

The radiation in space is not dangerous because it just produces a bit of current that can flip a bit; it is dangerous because it is strong enough to bump a bunch out of their position in silicon crystal, causing permanent damage that will accumulate over time. And when your feature size is starting to measure in <100 atoms it would be easy for single high energy event to break whole core.

The modern chip is more power effect by using traces that are closer together and thinner.

In the context of radiation hardening by design, this means that a cosmic ray strike could evaporate part of the 5nm trace or cause arcing to the next trace.

A lower voltage chip can be more easily permanently damaged by a cosmic ray. That's why the RAD750 has its radiation resistance measured in grays.

> The gray (symbol: Gy) is the unit of ionizing radiation dose in the International System of Units (SI), defined as the absorption of one joule of radiation energy per kilogram of matter.

This is the total absorbed dose - not the flux.

Additionally, this is for the entire system - not just the CPU. It's not just "here is a redundant CPU" but also "here is the entire system that the CPU is part of" that is redundant.

The JWST is using a RAD750 with 44 MB of ram... and yea, that's not a lot, we're not trying to ship super computers out there. The alternative that you're suggesting is suggesting putting those all in multiple out in space and checking them against each other constantly while the system is in use collecting data.

The power budget is slim and also coupled to the heat dissipation budget. The JWST has a tighter heat dissipation budget than power budget... and the Perseverance has a tighter power budget (total rover power budget is 110 watts).

The challenge I'd pose is "what other processor arrangement fits within its power budget?"

I'm also going to point out that I got the numbers wrong for Perseverance's power budget for its CPU. The single board is 10 watts, the cpu is 5 watts.

https://en.wikipedia.org/wiki/RAD750

> The CPU can withstand an absorbed radiation dose of 2,000 to 10,000 grays (200,000 to 1,000,000 rads), temperatures between −55 °C and 125 °C, and requires 5 watts of power. The standard RAD750 single-board system (CPU and motherboard) can withstand 1,000 grays (100,000 rads), temperatures between −55 °C and 70 °C, and requires 10 watts of power.

https://mars.nasa.gov/mars2020/spacecraft/rover/brains/

> Unlike people and most animals, the rover's brains - its computer - are in its boxy body. The computer module is called the Rover Compute Element (RCE) - there are actually two identical RCEs in the body so there is always a spare "brain."

Also - https://space.stackexchange.com/questions/50470/what-makes-i...

These are modern chips that are designed for that efficiency - it's just that there are other constraints too. You are unlikely to find a lower wattage processor that you can run with redundancy that is able to tolerate space travel.

https://link.springer.com/article/10.1007/s12567-016-0138-0

Radiation Effects and COTS Parts in SmallSats https://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=2...

Note that the cubesat has an average lifespan of about a year.

i guarantee the people building the spacecraft and choosing the CPUs have thought about this more than you, and have much more knowledge about the problem space than anyone here questioning their decision-making processes.

if it was a good idea to have multiple redundant CPUs on these spacecraft, the spacecraft would have multiple redundant CPUs.

I will note that they do have multiple, redundant CPUs... they just aren't all active at once.

https://www.jpl.nasa.gov/news/press_kits/mars_2020/landing/m...

> Just like Curiosity, Perseverance has two main computers, or “brains”: One is active at any given time, while the other serves as a backup. Both use radiation-hardened RAD750 computers, and together, they’re referred to as Rover Compute Element A and B.

> Another RAD750 serves as a third “brain,” on Perseverance. Called the Vision Compute Element, it is equipped with a special card for analyzing images. This card has a Virtex-5 field programmable gate array (FPGA) and is the driving force behind Terrain-Relative Navigation and the Lander Vision System, which analyzes images of the landing site during descent and compares them to an onboard map to determine the rover’s position relative to the ground.

That "Virtex-5 FPGA" is an interesting one too...

https://www.xilinx.com/products/silicon-devices/fpga/virtex-...

Where it is listed as "Space-grade Virtex-5QV FPGA"

I said I do not buy the power argument and I received two high quality comments explaining issues beyond power consumption. Your appeal to authority adds nothing of value.

Power arguments is interesting. Triple redundant 5W processors and 5W more for the redundancy support circuits(must not fail)? Sounds cheaper and easier to just slap on that 20W part, unless that 20W part is so expensive that it outweighs engineer man-hours for the 5W x 3 solution, but I can't see why it would unless said engineers are massively under-paid.

I'm going to correct a previous part - the 20W appears to be the budget for the system.

The RAD750 is a 5W processor and on a motherboard, is a 10W system.

https://en.wikipedia.org/wiki/RAD750

> Quad Core i5 sitting in a 13-inch MacBook Pro 2018 is a 28 Watt processor. It can heat its thin aluminum chassis to really high temperatures up to a point where it becomes an issue for some users.

Worth nothing here with Intel's Turbo Boost stuff this processor will draw a LOT more than 28W during normal operation - which makes this an awkward comparison.

> couldn't NASA use these for redundancy?

> how much more resilient are older processes with larger feature size compared to modern 6/7nm ?

If I may paraphrase your questions, I think you are trying to ask "isn't rad hardening a bit overrated? are space chips that good?" - and a Wikipedia edit[1] that has kept most of its original form for more than 15 years have been answering that question thanklessly:

> Hardened chips are often manufactured on insulating substrates instead of the usual semiconductor wafers. Silicon oxide (SOI) and sapphire (SOS) are commonly used. While vanilla chips can withstand between 5 and 10 krad, SOI and SOS can survive doses many orders of magnitude greater.

So the answer is, if 10krad(in lifetime) and [unspecified] events/seconds is enough for your requirements, maybe you don't need rad-hard chips. If your requirements are going to be magnitude grater(you can math it from data), then you might want it, and yes, there are differences in orders of magnitudes.

1: https://en.wikipedia.org/w/index.php?title=Radiation_hardeni...

>With modern CPU's having so many cores and threads being so cheap, couldn't NASA use these for redundancy?

Nasa needs redundancy. Say we ran a program in triplicate, one on each core, and compared the outputs. We still don't have redundancy here because the chip can still fail in a way that will cause problems for all three cores.

>My second question, how much more resilient are older processes with larger feature size compared to modern 6/7nm ? Compare modern with older (14nm/22nm finfet, 28nm/32 soi) and ancient (65nm or larger).

Larger gates tend to be more resilient to radiation. I don't know exactly why. The fabrication tech is also important. Some types of chip manufacturing processes results in more resilient chips. I'm sorry but I don't know enough about it to elaborate further.

Not an expert but essentially radiations can produce electrons in random places flipping one or more transistors unexpectedly.

Larger transistors need more electrons so they are slightly more resistant.

Because larger transistors need more electrons, they also need more power to switch. I guess that circuitry made from valves/vacuum tubes might provide high radiation resistance?

I mean, you could just use big transistors

That's single event upset (SEU). The other radiation issue is total dose. That's when charge gradually builds up in insulating gates due to ionizing radiation. It's permanent, and eventually reaches the point that the transistor is always on so has a permanent fault. One of the factors is collection area, so larger gates are not less susceptible because they have a larger collection area. But if they run at a higher voltage that can make them more tolerant to total dose.

THe other problem is that it can bump the atoms permanently off the silicon lattice, so permanent damage. Bigger transistor can just survive more hits

> With modern CPU's having so many cores and threads being so cheap, couldn't NASA use these for redundancy?

Bold No.

CMOS technology have one drawback - it is prone to latch up by ionization, because it have The parasitic p-n-p-n thyristor (it is locked in wrong state, to reset must do full power off).

Rad-hardened designs have to deal with this, they used special type of die material and clamping diodes, so their technology essentially different from civilian designs.

> With modern CPU's having so many cores and threads being so cheap, couldn't NASA use these for redundancy?

Maybe? But CPU cost basically doesn't matter compared to launch cost. Power budget given the required computing needs during critical sections of the mission is probably the biggest priority after reliability. Modern cpus are good at compute/watt, but not necessarily at low watts at low load.

No idea re: feature size, for cosmic rays and dram, it was thought that shrinking beyond some point was going to increase failures dramatically, but then it was fine. But in space radiation is more than cosmic rays.

> With modern CPU's having so many cores and threads being so cheap, couldn't NASA use these for redundancy?

Think shuttle had 3 separate computers running same code and having mechanism to vote out the one giving bad output ?

There are also some automotive ARMs that are basically 2 cores shifted physically 90 degrees running lockstep and some logic to reset if they disagree

> My second question, how much more resilient are older processes with larger feature size compared to modern 6/7nm ? Compare modern with older (14nm/22nm finfet, 28nm/32 soi) and ancient (65nm or larger).

I'd imagine by a lot. Think of the charged particle basically a pulse of charge. The bigger transistors are the bigger capacitance are and the more charge you need to flip anything. Any potential damage it could cause to the chip would also increase. I didn't found anyone trying to make "same" chip in significantly different process.

Reply on news.ycombinator.com