CPU manufacturers do do huge amounts of testing, and Intel does formal verification of some functional units. The reliability is far better than most software, in part because making a new release costs billions.
I've never seen Intel do a very good job at failure analysis or following on with failures unless prodded very hard.
* For Intel that would be companies like Dell, Apple, HP, and maybe a couple of others.
So in that super rare case of actually running into a CPU defect, it's a mindfuck, it'll drive you crazy. You'll be looking for the flaw in your algorithm which makes it fail once a week under production load. But you just can't find it, it makes no sense ...
(When it comes to drivers for network/storage/graphics etc devices, it's a whole different story. Those things are piles of bugs that need work-arounds in drivers.)
Or on the network level, a VPN that failed only when traversing one possible route between company offices.
The symptom was that a board with a specific microcontroller on it would be working fine, then after a power cycle it might not keep working. A flash dump would show that the reset vector, the first byte of flash on that system, would be all zeroed out. Of course the system would not run anymore, but why did it happen? After months of intermittent debugging and trying to reproduce the cause was determined. At least under certain conditions the brownout detection level was lower than the voltage level that caused the CPU to make errors. If the board lost power slowly then the CPU would start executing corrupted / arbitrary instructions which generally included lots of zeros. It would occasionally write zeros to the zero address, bricking the board.
Since then we have external power monitoring and reset circuits on all the new boards, but existing ones needed a fix. Luckily the board had power failure interrupt connected, so when that triggers we reconfigure the CPU to execute on the slowest possible clock rate, which greatly reduced the occurrence.
"Don't use the bookshelf over there, physics is broken on that shiny spot." :D
Modern day VM software has various levels of exception checking, and code to catch/mitigate/etc when bugs crop up.
So, a universe-capable simulator might have any kind of behaviour if/when a bug occurs. It doesn't need to be an unbounded, runaway scenario. :)
It probably happens more often than we imagine ;-)
> "Don't use the bookshelf over there, physics is broken on that shiny spot." :D
You know "The Animatrix - Beyond"?
https://en.wikipedia.org/wiki/The_Animatrix#.22Beyond.22
Seems like the same kind of concept. :)
// Workaround: Make the accessor method return fuzzy values for either of those values.
If Intel had to completely disable hyperthreading in Skylake and Kabylake that would make the premium anyone paid for i5 vs i7 worthless.
despite what cpuinfo tells you, no HT in i5.
and my previous comment was ironic :)
Or has that changed? At one point, i7 was full-featured, i5 was an i7 with HT disabled, and i3 was i7 with HT intact but smaller caches. Is that different with Skylake/Kabylake?
and my previous comment was ironic :)
Ack - sorry. I must be irony-impaired. That's why I don't post very often. :-)
yes it happens, but software bugs happen every day where as if your system blue screened every day you know dang well you'd be on the phone with the hardware vendor for a refund / new system.
Well, since hundreds of millions of people use Skylake/Kaby Lake CPUs for 2 years now, and only now we learn about this, obviously this is not of the "system blue screens every day" variety but a very rare bug.
Not to be anal but we can't know this.
I think we're getting to levels of complexity where the process Intel uses, with lots of different QA and testing teams doing their best to look for bugs, just isn't going to cut it. We need formally verified models transformed step-by-verified-step all the way down to the silicon. It's already feasible, with free tools, to formally verify your high-level model (using e.g. LiquidHaskell) and then transform this to RTL (using e.g. Clash). With Intel's QA/testing budget, it's well within reach to A) verify the transformation steps and B) figure out how to close the performance gap between machine-generated (but maybe slower) and hand-rolled (faster, but evidently wrong) silicon.
but Intel isn't allowed to have issues in their processors, even if they can fix them with a software (microcode) update?
They could've caught and fixed this one with some more testing, before actually releasing. Formal verification isn't necessarily going to help, if it's a statistical type of defect --- thus the concept of wafer yield, and why dies manufactured from the exact same masks can behave completely differently with respect to the clock speeds attainable and voltages required.
I believe the main issue with formally verifying everything is that reasoning about parallel code is extremely hard and might well make the entire endeavour unattainable. It would be great to have formally verified CPUs, though.
> They could've caught and fixed this one with some more testing, before actually releasing.
Isn't this true of any bug?
I know they did AMT but that was a special case and different.
Therefore, CPU designers verify the important units (e.g., the ALU) independently of each other, and then try to verify their interaction. But a system level design verification check is simply not possible.
Obviously, faults that result from fabrication can still take place, so they are tested for using BIST and JTAG. Test coverage can be pretty high, but obviously not 100%.
As you can see, there are still a ton of places where hardware errors can seep through.
EDIT: My wording was poor, but what I meant was that it's not like hardware vendors do not make bugs happen due to deficiencies in their design, which are not only tolerated, but often reflected in software, which runs counter to OP saying that apparently hardware vendors are not allowed to have bugs, but the software ones are:
> So it's okay for software to have bugs that get fixed (I think everybody here acknowledges that software will always have bugs), but Intel isn't allowed to have issues in their processors
Ever since I moved from C to Java (I like low level stuff but the company I joined is such) I have been having one of three major problems - logic bugs, too many frameworks, causing bugs due to lack of in depth understanding of each one of them, and GC bottlenecks. Of all of them, I hate the GC ones the most.
I don't think this is sustainable, and I think that massive arrays of relatively simple processors. First, we will need a culture shift that learns programmers to program concurrent programs from the start, and this will take a lot of time (because technology moves a lot faster than culture).
http://techreport.com/news/26911/errata-prompts-intel-to-dis...
The TSX situation was indeed unfortunate, I have one of the affected CPUs. But it was a new feature that was broken, not something that used to work, so the bug's impact was less serious, and disabling the feature didn't have too much of an impact.
The Dell XPS 9350 (Skylake) has numerous issues with GPU noise, USB-C compatibility, wifi reliability - Dell don't care unless your consumer laws are strong enough to make them care.
For example, you can not possibly formally verify the fetcher unit on its own, because the state space that you need to cover for several cycles for all the module inputs and outputs is beyond the capability of any formal verification tool.
Typically, you run formal verification on sub-blocks of sub-blocks of FUs.
For this particular bug, it looks like multiple functional units are involved, so it might have been missed by formal verification.
Then use a proof methodology that doesn't require exhaustive enumeration. This objection is actually fairly alarming to me; perhaps there is a larger disconnect between industrial formal techniques for hardware and software than I thought. Strings also occupy a "large state space", but this obviously doesn't prevent us from doing formal verification on functions over strings.
Then use a proof methodology that doesn't require exhaustive enumeration.
Strings also occupy a "large state space", but this
obviously doesn't prevent us from doing formal
verification on functions over strings.
Ok, so what this tells me is that you're not aware of what modern verification techniques look like.
There's not really any one resource I can point you to, but take a look at these links. I've used these or similar technologies personally, but there are others I haven't used.
https://en.m.wikipedia.org/wiki/Intuitionistic_type_theory
https://en.m.wikipedia.org/wiki/Homotopy_type_theory
https://leanprover.github.io/about/
http://goto.ucsd.edu/~rjhala/liquid/liquid_types.pdf
> What is your reason to think this is 'obviously' the case?
Because I've done this, and anyone who claims to be familiar with formal methods should be at least passingly familiar with all the things I mentioned.
Even if you aren't, if you've ever heard of SAT (a fairly universal CS concept) you should at least be familiar with the idea that non-exhaustive proofs are a thing.
Ok, so what this tells me is that you're not aware of what
modern verification techniques look like.
Notwithstanding your example that it's possible to non-exhaustively verify some functions on strings. There is a quite some distance between that to verifying just any function.
Because I've done this,
I think this may be an example of the disconnect between research and the industry. Researchers say things are solved when they have shown something is possible and published about it. They feel it's then up to the industry to extrapolate, while research moves on to exciting new and greener pastures. Meanwhile the industry thinks the results are too meager, thinks the extrapolation involves a lot of technical difficulties and generally is not willing to spend enough money on what they cannot see as anything but a longshot.
If you do any model checking at all, with tools like SPIN or TLA+, you are already in the state-of-the-art minority in industry.
If done naively, it doesn't work because you are unlikely to hit the problematic bug. So you have to be clever: do not generate inputs uniformly! Generate those inputs which are really really nasty all the time. If you find a problem, then gradually simplify the case until you have reduced the complexity to something which can be understood.
With some experience for where former bugs tend to live, I've been able to remove lots of bugs from my software in edge cases via this method. (See "QuickCheck").
Formally verifying something like a multiplier block is difficult but doable if you care. Formally verifying an FPU is probably at about the limit.
If you want formal verification, you would have to simplify a modern microprocessor a lot.
Perhaps we are imagining different formal verification methodologies. Can you tell me what kind of formal verification you're referring to?