Intel updates mysterious ‘software-defined silicon’ code in the Linux kernel

For Ryzen CPUs (ones without an iGPU), they don't fuse it off, and as such it can unofficially work.

For Ryzen APUs (parts with an iGPU), they fuse it off, and as such it will never work there.

The parent comment saying that they never see ECC errors in the wild is missing a few things:

- Server memory tends to be clocked lower than consumer memory, so errors are less frequent to begin with.

- The errors are not evenly distributed. Some memory sticks have a high error rate, others are virtually zero. There's batch-to-batch variations.

- I've done my own tests on hundreds of servers. We run burn-in tests for about 24-48 hours. About 95% have zero errors of any kind, but 5% have a high enough rate that putting them into production would be a mistake. ECC allows us to catch those bit errors instead of silently accepting them and allowing data corruption to creep in.

- I've personally had 3 different personal computers experience high memory error rates, to the point of multiple BSODs per day and data corruption. They all started off "good" and slowly turned "bad". The only reason I knew to look for memory corruption as the root cause is because of my extensive industry experience. A grandma using the same PC would have just blamed Windows for being unstable.

- Vendors like Microsoft simply ignore all crash error reports sent back by telemetry with only 1 or 2 samples, because those are virtually guaranteed to be caused by memory corruption, not programmer error. I've done similar memory dump collection and found that easily 30% of all crashes were unique in this way, suggesting that ECC memory could improve PC stability significantly.

- Suggesting that ECC memory is not needed because "good" memory doesn't need it and only "bad" memory is a problem is missing the point. All memory is bad, it's just that the bit error rates are different!

It's true that ECC will catch errors caused by cosmic rays, but in practice most memory errors are just from faulty memory. Large studies in Google datacenters showed that the errors were heavily concentrated in a small number of DIMMs: http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

The catch is that you don't really know if you have one of these faulty memory sticks. I always run memtest86 overnight to check for obviously faulty memory, but some of these errors could take months to manifest.

I had a laptop with a RAM cell that failed. It would show up in memtest86 in a matter of minutes, yet surprisingly I didn't notice the issue for a very long time in day to day usage. I always wonder if there were random bit flips in anything I worked with during that period, but I'll never know.

With ECC, it's just one less thing to worry about.

There are some bugs that get raised on linux once every few years that are so obscure they are suspected to be random memory errors.

Without ECC any of the above will crash an app, or even the entire machine. With ECC you'll get a log message, and if it's a single bit error it will be automatically repaired. If it's not fixable and it's in userspace that application is killed (and the kernel logs the error), if it's in kernel space the kernel will panic.

So generally it makes your machine more reliable, and easier to debug. Generally repeated errors are a fault of some kind and you can easily tell which dimm it is. Without ECC you end up troubleshooting all causes of crashes, and even if you know it's memory, you can't be sure which dimm it is.

So I think it's well worth the minimal premium to make your machine more reliable, and if it's unreliable it's much easier to track and fix the problem.

Amusingly without ECC heavy CPU use with parallel gcc compiles causes a particular error. Not sure if it's in the FAQ, but it's well know that the particular error means you have a CPU/RAM problem, common with memory errors, CPUs that are too hot, or overclocked CPUs.

But these weren't just cosmic-ray-induced errors. The DIMMs with errors were far more likely to have more errors in the future, pointing to faulty memory cells.

Where? Are you talking DDR4-3200? These are 40-50% higher than non-ECC UDIMMs. It usually gets worse as capacity increases. Additionally, there are only one or two models of 1x32GB UDIMM in production. These sticks are quite rare. You don't just go to Newegg or Amazon unless you want to pay out the arse for them. And in addition to that, ECC has worse timings which are important for Ryzen.

> Registered DIMMs have a higher premium

You have this backwards. RDIMM are common and cheaper because that's what servers generally use. UDIMM ECC cost more because, well, they barely exist as a thing.

The likelihood that a RAM error is corrupting data silently is slim, given the same corruption could affect the RAM storing the OS kernel or your application stack.

If you have bad non-ECC RAM, you will notice issues that warrant further investigation (i.e. memtest). But that assumes you're savvy enough to understand what failing memory looks like.

I'm going to claim it wasn't entirely my fault. ARCnet did automatic resends for you so if fixed what the CRC detected, and silently what it didn't detect through. I had no idea the errors were happening.

ECC is nice, but as you say I don't need fancy hardware to correct a malfunctioning RAM chip. I can do that perfectly well with a hammer. What I need is the hardware to tell when and where to deploy the hammer. Things were actually better 30 years ago - back then PC RAM had parity. It's cheap, fast and effective enough. I have no idea what went wrong.

You bought a 60 kWh car. If it performs as advertised, you not entitled for more, no matter what hardware is inside. And if you find a way to exploit the full capacity of the battery, you can, it's yours, but don't count on Tesla to support it.

I have no problem with market segmentation. It allows more people to have access to a product while protecting the margins of the manufacturer. Because, without margins, there is no product.

And in fact, I am happy when I am traveling and the student next to me paid half of what I paid, I could afford the higher price ticket in exchange for more convenience, so no problem for me, but if the student had to pay the same price as I did, he wouldn't have been able to travel at all. I was like him years ago, and our children will probably be too.

I understand in case of Tesla you don't like the idea of hauling those extra unused cells around. But you are buying a car model with a specific mass, acceleration and range. It makes no difference whether how exactly the producer fit into those parameters.

That's definitely not how physical property works.

There's literally tens of thousands of devices for the PC platform and Linux has no problem supporting them if the hardware can be documented. Why can't the vendor document that instead of hiding this stuff behind opaque binary blobs that can have security vulnerabilities? Whether this is ARM or x86 doesn't matter.

> ACPI OTOH allows random x86 board vendors to literally have hundreds of products themselves that not only boot linux, but a pile of other OS's because they aren't wasting their time writing drivers for every single board and voltage regulator.

I'm parsing this as "ACPI OTOH allows random x86 board vendors to develop shitty hardware, cut corners on documenting it, and not care as long as Microsoft OS's boot on it."

> they aren't wasting their time writing drivers for every single board and voltage regulator.

Document the damn registers and hardware interfaces and then someone out there will write it for them.

> Its also largely the reason why linux works on 20 year old PC's that no one is actually testing on anymore.

Well Linux still supports hardware that predates ACPI, like floppy drives, so this is making a connection where there is none.

> The alternative is a monoculture where the HW is basically provided by a single vendor that doesn't change much and provides a half dozen supported configurations.

Wrong. Before ACPI, for example, there were many chipset vendors--Opti, VIA, etc. ACPI didn't kill these off but there wasn't a monoculture before ACPI.

> I would point to the zseries here

So ... do we want an IBM-mainframe like monoculture where you have to depend on IBM when the hardware changes?

> allows IBM to change the underlying HW details

The job of abstracting the hardware details is the operating system and operating system drivers. Literally that's 50% of the reason why you have an OS in the first place is to abstract I/O into things like open(), read(), write() or other interfaces that are portable because of underlying drivers. If the hardware is so different that existing I/O calls can't handle it, you need to develop new ones - that's what happened with Berkely sockets - NICs are not block or character devices.

If the drivers are open source then they are bug-correctable and useable even well after the hardware manufacturer goes away. Embedding that in closed-source platform-firmware makes you dependent on that platform manufacturer. A strong contributor to the Wintel monopoly.

Some good ideas work. Some good ideas don't. Doesn't mean they were bad ideas, only that unknown at the time factors or future developments made them bad ideas.

Somehow nobody shed a tear for the poor enterprise customers until miners came along. But that's what it is, miners are enterprise customers, they use them to make a profit.

Similarly, almost all R5 consumer processors are made by artificially disabling fully functional cores on R7 chiplets. AMD already had nearly 80% of the chips coming off the line with 8 functional cores at Zen2 launch, and numbers have only gotten better since there. Yeah, maybe a few fail clock bins or other things, but those bins are chosen very loosely such that they're not throwing away any significant amount of chips anyway. The overwhelming majority of R5s are perfectly good chips being gimped and sold as lower chips, because the availability and demand are exactly opposite of each other - they have the highest availability of 8-core chiplets and the highest demand for 4- and 6-core chiplets. How do you square the two? Not by lowering prices of your high-margin products - you keep those high to extract the maximum price from price-insensitive consumers, and you artificially gimp a bunch of them for the rest of the market to protect your profits in the higher segment.

https://en.wikipedia.org/wiki/Price_discrimination

The thing to remember is that home users are almost always the beneficiary of market segmentation, because we are the most price sensitive. The alternative to Core-vs-Xeon-E segmentation isn't that everyone gets Xeons at i5 pricing, it's that the i5 now costs almost as much as a Xeon. The R&D has to be paid back one way or another, or else R&D will be significantly reduced (most likely meaning much longer product generations and slower moves to new nodes, etc). Price discrimination means enterprise customers pay more than their share and home customers pay less than their share, so if market segmentation goes away and those equalize then home customers will have to bear more of their fair share of the product cost.

Again, who wants to race to pay more for their gaming graphics card so that Boeing and cryptominers can pay less? It would be great if everything were free, or priced at its true cost, but in the real world R&D needs to be paid for somehow.

Oh, and the other thing is Enterprise customers actually love this, it's called Capacity on Demand and I'm sure the reason Intel is doing this is because their customers keep asking for it. Sun, Oracle, Fujitsu, IBM - all the big enteprise vendors will sell you a machine with a bunch of cpu and memory that you're not allowed to use or touch! But if your workload scales up, instead of having to shut down your mainframe to upgrade it, you can just pay money and turn the cores on! And sure, it would be nice if all software were just microservices that you could re-instantiate onto another machine, but some stuff (like databases) can't be trivially scaled and partitioning (read: stale data reads) can't be tolerated. The only surefire answer to CAP theorem is One Big Machine such that you never have to deal with splits in the first plac, and that's what some situations need, and they love this.

Was the car sold to its original owner with those features enabled? If so, it's effectively the same as the original story reported, meaning that Tesla expected to be paid twice for the feature.

You don't want to get letters from Oracle, IBM, Intel and Co. because you infringed some of their EULAs. They may be invalid, but it will be a very costly decade long court case. And the other part will pump thousands of times more money into it than you can afford, because when they loose they will loose a big income stream in the future.

It's really stupid that other vendors can't just do the same.

I assume you meant to be a help by assuring people their computers aren't backdoored, but I don't think you have the certainty to claim ME is sound and secure. It just reminds me of how well-meaning people insisted the USG wasn't interested in capturing everyone's communications, back before it was accepted fact, pre-Snowden.

https://www.youtube.com/watch?v=jmTwlEh8L7g This is one of many amazing videos this guy did before they hired him.

Perhaps funny, but inevitable due to the lack of knowledge. Why don't you do us a solid and get us some documentation for our hardware using a cell camera?

...and hardware manufacturers make money selling hardware, so why keep the details of how to use it secret? In the days of SDRAM and DDR(1) a lot of companies, including Intel, were far more open about documentation than today.

I've analysed the init code from BIOSes and it is indeed not that complicated.

If they can sell another 16 cores at no incremental cost post sale, that's attractive. As long as it doesn't prevent that customer from buying a whole new 48 core chip from them, anyway.

Chips have been known to have perfectly functioning blocks disabled, or frequency limited not for improved yields but simply because the demand for a lower end SKUs was so high. Unlocking them was a matter of penciling some fuses, flashing a firmware, or simply changing a setting in the BIOS.

If anything this kind of licensing opens the door to "hardware piracy" where under the correct conditions you may be able to hack older, or unsupported hardware to gain the full functionality.