Cores that don't count | Hacker News Reader

Cores that don't count

muratbuffalo.blogspot.com

61 pointscanthandle30 comments

Detection is certainly the harder part - but the rest of it is pretty well-trodden ground. Some POWER9 users noticed that linux was under-reporting their core count and further investigation showed that the CPU's RAS had identified some cores that were acting a little funky - so it automatically dropped them into a guard partition and logged the action in the persistent circular buffer that nobody had bothered writing open source code to watch. Of course IBM had already written a ton of software that had it covered in AIX. I've noticed the same thing on a lot of other platforms, I'm looking at an HP MicroServer right now that has low level hardware error logging that never makes it past the bios firmware - and I've been meaning to fix that for about two years now. It might be a good idea to tackle that issue before we start getting too clever with solutions.

Dynamic reconfiguration: Basic building blocks for autonomic computing on IBM pSeries servers

https://sci-hub.se/10.1147/sj.421.0029

Cores that don't count = floating-point coprocessor cores? :-)

But seriously, though,

> I think fail-silent CEEs is weaker than the adversary Byzantine failure model.

Of course they're weaker than byzantine failures. There's time locality, and the failure in themselves are not particularly hard to detect if some other core checks the results (although that obviously doesn't happen after every single computation).

wait... consensus algorithms exist? can they work for human brains?

experiment :

tell a bunch of people that if they form consensus about, say, a color, they all get $5 (or whatever). have them attempt to reach consensus only using the exact mechanisms of a consensus algorithm. (research what they are)

You need to specify further environment parameters: Is the communication channel lossy? Can all the participants be trusted? Is there a time limit?

behind the scenes, someone offers one of the participants $10 to prevent consensus.

You mean if people perform a successful vote on something they don't care about, you'll give each $5? You don't think anyone could do it in under a minute?

Yeah but like not face-to-face. Using the algorithm, but running on humans. Like the old human computers.

All you need is for one person to propose something and a majority to say they agree? Is that what you mean?

The conclusion is an utter joke: “Maybe this will lead to abondonment of complex deep-optimizing chipsets like Intel chipsets, and make simpler chipsets, like ARM chipsets, more popular for datacenter deployments.”

Why is that a joke? There's a reason why AWS is pushing Gravitron so hard (and they are pretty damn fast)

Why would one presume that the reason has anything to do with computation error rates, rather than something obvious and mundane like "Graviton instances are more profitable"?

Controlling the entire stack, from control plane to core layout is absolutely better for isolating computation error rates if they occur.

Probably because datacentre ARM chipsets aren't significantly simpler than Intel ones. Not in any ways that would reduce this problem anyway!

The article even says this is because of reduced feature size, not chip complexity.

Maybe we should run the code twice on different cpu architecures and only accept the results if they are identical. I’ve heard of high reliability environments doing this, but maybe there are cases for doing it in web/IT as well.

This would help catch a large variety of possible errors, including but not limited to cpu bit flips, cpu bugs, memory errors.

Maybe, but in most situations you're probably better off adding some more developer time to work on software bugs than you are doubling your computation cost.

This kind of problems we're discussing and software bugs are not the same thing. A memory bit flip is not really a software bug is it

Of course they're not the same thing. But they both make things go wrong. There are situations where hardware bugs are important, and there are situations where hardware bugs don't make a meaningful difference.

In my experience, achieving perfect determinism across two architectures is a problem that cannot be easily hand waved away.

For 99% of cases it is fine, but when you run into that last 1% it hurts.

It might be a huge hassle but it might be worth it if high reliability is desired.

Triple modular redundancy - explored on the earliest computers, when these kinds of CPU errors were common (but for other reasons). It isn't commonly implemented today for the same reason that ECC RAM isn't in everything by default - people tend to prefer paying less and just crossing their fingers.

Rad-Tolerant Lockstep Quad Core 8051:

https://www.eetimes.com/rad-tolerant-lockstep-quad-core-8051...

Tolerating rads with cold-war mcu softcores. Glorious.

Previous discussion:

https://news.ycombinator.com/item?id=27378624

https://news.ycombinator.com/item?id=27408398

> A deterministic AES mis-computation, which was “self-inverting”: encrypting and decrypting on the same core yielded the identity function, but decryption elsewhere yielded gibberish.

This is such a unlikely thing to happen that it is likely a fault in software that is used to validate this.

all it takes is something that reliably bitflips a key holding register

What needs to happen is to bring ECC to all levels of a chips logic to solve this. ARM vs RISC-V vs x64 doesn't address the problem as nothing in them inherently solves the problem. Making adders that add with ECC will.

If all of the logic also operates on ECC with the data, chip yields will also be improved. Say an core of the chip only produces the correct result 99% of the time, currently you have to disable that core. With ECC logic, you can still use it, as it doesn't matter if it has an additional 1% chance of a bit flip, as all of your logic is now immune to single bitflips. For mission critical logic/applications, one can scale up the ECC so its immune to more bitflips before an error is introduced.

What leads us to believe that there is not already fault detection in execution units? We have really no idea what's going on at the gate level in CPUs.

i know Intel and AMD like to hide hardware features - but given the design cost and the non-negligible overhead..one would suppose that they would at least surface a counter in the documentation..and probably even...you know, market it as a feature.

Is there a way to add ECC into an ALU without making it massively slower?

I’m fairly sure you can’t do that. For example, suppose you have logic that uses some inputs to compute an output:

  A, B ⇒ C

Add ECC bits to the inputs, and you want

  Aa, Bb ⇒ Cc

Now, if you want this to detect errors made by the “⇒” part, you can’t do this as “drop the ECC bits, compute the result, compute the ECC bits of the result”.

So, how do you compute the ECC bits from only the Aa and Bb bits without having to compute the C part?

Depending on the ECC logic chosen, that might be doable for bit shifts, but for addition? For multiplication? For IEEE float square roots?

You would have to develop a ECC that worked similar to homomorphic encryption, where you can do computations on the chipertext (or in this case, the ECC) without knowing the plain text. For this application, plain text would effectively be the int/float you are doing math on.

Since its possible with crypto, I don’t think it’s an insurmountable problem to create an ECC code where

Ecc(a+b)=Ecc(a)+Ecc(b)

It will likely not be as bit efficient as ECC without that property. Checking the ECC would also add a bit of overhead, so at first you would want to only check on a store instruction where you have to wait to select the RAM page anyways.

“I don’t think it’s an insurmountable problem to create an ECC code where Ecc(a+b)=Ecc(a)+Ecc(b)”

It wouldn’t, but that ECC wouldn’t work for multiplication instructions, square roots, etc. Dropping floating point and multiplication instructions will correct that, at the cost of significant speed.

I also think any such ECC effectively would be a copy of computing a+b modulo some constant you can pick.

If so, we’re effectively back at “compute the result twice, trap if the two results aren’t identical”

Reply on news.ycombinator.com