Is the x86 Architecture Sustainable? | Hacker News Reader

Is the x86 Architecture Sustainable?

robert.ocallahan.org

109 pointsAshleysBrain101 comments

Even ignoring CPU die size cost, there's another cost to legacy features: some of those features have register state, and contexts switching it is expensive. Simpler architectures can do crazy things like switching between threads in mere tens of cycles. X86: not so much. The OS needs to deal with DS, ES, FS, and GS, and the microcode needs to deal with CS, SS, and FLAGS. The segment registers are almost useless these days, but saving and restoring them is very slow.

Even x86's model of handling CPL (privilege level) isn't ideal. I've tried to push RISC-V toward a model where user and kernel code have entirely separate address spaces, which would let privilege handling move to the front end and potentially allow pipelining across privilege changes. Amusingly, x86's privilege mechanism is complicated enough that Intel implemented it wrong in a few CPUs with MPX, resulting in a bizarre erratum.

I would love to see x86 add a "clean mode" in which a lot of legacy features are inaccessible to user code. Then kernels could try to run programs in clean mode and, if they don't fault, context switching between clean mode threads would be much faster.

Even x86's model of handling CPL (privilege level) isn't ideal.

Because each was retrofitted, each protection ring in x86, from -1 to 3, works differently. When IBM originally invented virtualization (IBM 360/67, late 1960s), they got it right; all the layers work the same, and you can run VM on VM on VM until you run out of memory. That's been continued into today's IBM mainframes.

I'm not talking about the fact that the privilege level is two bits instead of one bit. What I mean is: the privilege level is more or less duplicated in quite a few different pieces of CPU state. There's CPL, CS.RPL, CS.DPL, SS.RPL, and SS.DPL, not to mention CR0.PE. The one that matters is CPL, but CPL itself isn't an architecturally visible register except (I think) on AMD systems with SVM -- it's merely a concept.

CS.RPL, CS.DPL, SS.RPL, and SS.DPL all usually match CPL, but I think that all four equalities have nasty corner case exceptions. This results in a bunch of overhead when you change privilege: the CPU needs to update CPL, CS.RPL, CS.DPL, SS.RPL, and SS.DPL. This means that you need to change two of these nasty large legacy registers, the values of which barely matter for practical purposes, every time you enter or exit user mode.

Saying there's a ring -1 is a red herring here. There's SMM, which is a mode of execution that's more privileged than ring 0, but it doesn't have a number -1 associated with it, nor is it part of CPL. Similarly, ring 0 is split into root and non-root when virtualization is in play, but that's also a bit beside the point.

[1] Check out the Linux commit message for ae9fedc793c4d98aa9bb298585b2b9246096ce65 if you want to see a bit of how nasty this is.

I'm not following the connection, IBM used more than two protections and thats why we have the two unused rings in x86? Can you elaborate?

> The segment registers are almost useless these days, but saving and restoring them is very slow.

Those are only saved/restored when switching between processes not between threads. Context switching when dealing with threads is typically done in software, not in hardware, precisely so that registers that are not used (like the segment registers) don't get needlessly saved & restored.

Saving/restoring registers isn't even the expensive part of a context switch anyway. Switching address spaces, cache flushes/misses, etc... are all more expensive than saving/restoring registers.

> Those are only saved/restored when switching between processes not between threads.

No, those registers are per-thread. The only things that are per-process are the page tables and, in special cases, CR4 and LDTR. If you want to see the code on Linux, look at switch_mm_irqs_off().

> Context switching when dealing with threads is typically done in software, not in hardware

Hardware context switches (the x86 "task" mechanism) have been obsolete for a long time and are unavailable in long mode.

> precisely so that registers that are not used (like the segment registers) don't get needlessly saved & restored.

And that's the problem: the OS doesn't know that the registers are unused until it saves them.

> Saving/restoring registers isn't even the expensive part of a context switch anyway. Switching address spaces, cache flushes/misses, etc... are all more expensive than saving/restoring registers.

Go compare WRMSR to MSR_FS_BASE to the cost of a CR3 (page table) write. You might be surprised. And that CR3 write is about to get considerably faster, maybe in Linux 4.13, when my PCID patch set lands. (Yes, Linux will eventually use WRFSBASE instead, but that's a minefield and will take a while.)

This person switches threads :)

But in practice, I guess this legacy architecture adds a nanosecond or two to a thread switch. Is that a big deal? How many switches does a desktop PC do in a second? How about an AWS/Azure/GCP server?

I've never experienced lag in a game or app or web server due to slow thread switch. Server side async is on the rise anyway, minimizing number of threads and switches.

> But in practice, I guess this legacy architecture adds a nanosecond or two to a thread switch. Is that a big deal?

I would guess the total wasted time is more like 500 cycles. This won't make your game lag, but it does mean that extremely low overhead synchronous IPC can't happen on x86.

What types of programs are impacted by the lack of low overhead synchronous IPC?

As an example, there are a bunch of things that work well on microkernels on non-x86 architectures that involve splitting drivers across processes.

On Windows, a lot of the windowing system is in the kernel in win32k.sys. Microsoft didn't do this for fun -- they did it because the old way (NT3.5?) was too slow.

Most registers including x86 segment registers need to be saved during context switches, regardless of its between threads or processes. E.g segment registers used for TLS. There are maybe optimisation for flat segments.

> Context switching when dealing with threads is typically done in software, not in hardware

I'm not sure any OS exists that does register saving in HW for switches between processes and in SW for switches between threads. Its especially doubtful for Linux, given it does not really have a very strong concept of what is a process vs. what is a thread.

> Those are only saved/restored when switching between processes not between threads. Context switching when dealing with threads is typically done in software, not in hardware, precisely so that registers that are not used (like the segment registers) don't get needlessly saved & restored.

You are incorrect. Context switching is done for threads in HW. I think you are thinking of cooperative multi-threading (like pre-Windows 95, or nginx or node.js or V8) instead of true context-switching threading.

All you need for preemptive multithreading is IRQ#0 to generate an interrupt. That's the entirety of it.

Preemptive multiprocessing then needs paging and such to make isolated memory a thing as well, but still doesn't need HW context switching.

> Context switching is done for threads in HW.

Uhm, x86-64 can't switch between tasks/contexts in hardware.

Nowadays CS, SS, DS, and ES are enforced to be 0 in 64 bit mode, leaving just FS and GS which are used for e.g. thread local storage

Not true. The CS, SS, DS, and ES bases are ignored in 64-bit mode, but those registers still exist, still carry their full, extraordinarily complicated values, and are very much observable. Even on AMD Zen, for example, SS handling is screwed up enough that every context switch has added overhead to work around a SYSRET glitch that would otherwise risk executing user code with a messed up bit in the "hidden" part of SS.

AMD should have killed these registers entirely in 64-bit user mode, but they didn't.

My understanding is that AMD probably originally designed SYSRET assuming that SS contained the correct values already (except SS.DPL of course, which I guess is where the stuff where they set it to RPL comes from). When they designed x86-64 however, they decided to set SS to null selectors in interrupts. Of course, these values are normally ignored in 64-bit long mode. Long mode is also when they fixed SYSCALL/SYSRET by adding an option to mask EFLAGS (and store the old value in R11): http://web.archive.org/web/20120813101603/http://www.x86-64....

BTW, SYSENTER/SYSEXIT was designed by Andy Glew, which mentioned in https://groups.google.com/d/msg/comp.arch/Bf-4VflljF4/4hNvyr... that "(I wanted to avoid changing SS:ESP, so that it could be just a single register write if CS:EIP were co-renamed, but microcode thought that would violate invariants about SS RPL they wanted to maintain."

I wonder if part of the problem is that AMD was still new to x86 CPU design back when SYSCALL/SYSRET was developed as part of the K6.

>"and the microcode needs to deal with CS, SS, and FLAGS. The segment registers are almost useless these days, but saving and restoring them is very slow."

Can you elaborate on whats special in terms of CS, SS and the FLAGs registers in terms of what the microcode must do?

>"Even x86's model of handling CPL (privilege level) isn't ideal. I've tried to push RISC-V toward a model where user and kernel code have entirely separate address spaces, which would let privilege handling move to the front end and potentially allow pipelining across privilege changes"

I thought that mapping the kernel into the top of of a userland process address space was supposed to be kind of elegant as it prevents a TLB flush. What is does "move to the front end" in your proposed alternative to this arrangement? This sounds interesting. Might you have any links regarding this?

> Can you elaborate on whats special in terms of CS, SS and the FLAGs registers in terms of what the microcode must do?

For SYSCALL, SYSRET, SYSENTER, and SYSEXIT, microcode needs to change the values of the registers (including the hidden parts). For entries through the IDT (interrupts, faults, and traps) and IRET, the microcode needs to go through a whole bunch of complicated logic including looking up the requested CS and SS selectors in the GDT or LDT.

> I thought that mapping the kernel into the top of of a userland process address space was supposed to be kind of elegant as it prevents a TLB flush. What is does "move to the front end" in your proposed alternative to this arrangement? This sounds interesting. Might you have any links regarding this?

Once upon a time, the x86 MMU, and probably most other MMUs, was quite simple. There was a TLB, which is essentially a cache that maps (VA) -> (PA, access rights, cache mode). A memory access looks up the virtual address (VA) in the TLB, then checks the access rights and accesses the physical address. In the event of the TLB miss, the CPU falls back to reading the page tables and fills the TLB. When you change to a different process with different page tables, you flush the TLB. You can either map the whole kernel into every process's address space of you can map a small trampoline that switches to the real kernel's address space and flushes the TLB every time you enter the kernel. From (IIRC) 486 onward, x86 would let you tag a TLB entry as "global", meaning it would survive a normal TLB flush and would only be cleared on a special type of flush. Linux uses global entries for kernel pages so that only user entries get flushed on processes switches.

Modern CPUs have a somewhat different arrangement. The TLB maps (VA, address space ID) -> (PA, access rights, cache mode) [1]. On a TLB miss, the CPU will look up the VA in the current process's page tables and add a TLB entry with the appropriate address space ID. On x86, the address space ID is (VPID, PCID) [2]. This means that you can sometimes switch between processes without a TLB flush -- you can change the page tables and the address space ID and leave the TLB intact.

I would like to see CPUs arrange it just a bit differently: have a separate address space, complete with a separate tag bit, for user mode and kernel mode. Then user accesses effectively look up (VA, "user", ASID) and kernel accesses effectively look (VA, "kernel", ASID). This would add almost no overhead. As an added benefit, when the front end decodes an instruction, it would know the whole TLB key (i.e. VA, user/kernel, and address space ID) and could decode user and kernel code differently. This means that kernel-mode instructions could be rejected in the front end. More importantly, if you imagine the user/kernel bit as being an extension of the VA, a transition between user and kernel mode is functionally the same thing as an indirect jump and could use all the optimizations, including pipelining, that indirect jumps use.

[1] It might be organized internally as (VA) -> (address space ID, PA, access rights, cache mode), but it doesn't really matter.

[2] VPID identifies the current VM guest, if any, and PCID is used at the discretion of the kernel. Linux may soon use PCID to allow a few processes to coexist in the TLB at once. grsecurity uses it for a totally different purpose.

Thanks for your detailed reply. You mentioned:

>"You can either map the whole kernel into every process's address space of you can map a small trampoline that switches to the real kernel's address space and flushes the TLB every time you enter the kernel"

Do you know what OS's employed the "trampoline" option instead of the more common mapping the kernel into every process address space? Were these commercial UNIX vendors maybe?

PGE was introduced in the P6.

Good job keeping me honest :). I usually pay attention only to what modern CPUs do and what the the oldest Linux-supported CPUs do. Which I guess means I should have known that 486 didn't have PGE.

Interesting. I thought they would do register renaming for thread switching as they do for speculative execution. Is this not the case?

I doubt a segment register is easily renamable. It implicitly affects most subsequent operations, and it's not really just a number. (You can find the full data structure documented in the virtualization section of the Intel and AMD docs because virtualization extensions let you read the registers in their entirety [1].). Also, the instructions that write to these regs have very complicated microcode.

[1] The AMD interactions between CPL, CS, and SS are so messy that there's still an unresolved crash when KVM runs an older Linux guest and exits all the way out to QEMU to handle, say, a virtio interrupt at the wrong time. QEMU tries and fails to read the state and write it back fully.

Also AVX512! And what the heck are we going to do when we need to save the AVX1024 register state! And we really going to push 1kb to memory per context switch? (I think this is somewhat solved by trapping when the registers are first invoked, and kernel drivers that use enhanced instruction sets actually need to pass flags before using them)

This highlights one of the advantages of context-switching in user space. The OS context switch can make few assumptions about the registers that actually need to be preserved once a register set has been used. Many user space context libraries, on the other hand, only preserve the half dozen basic registers per the ABI by default (very fast) and let the programmer optionally preserve the other register sets. In the vast majority of cases, you never need to preserve any additional state at the logical context switch points even when you are using those additional registers in your application.

Alternatively juat grep the function source and ban self modifying code. Or run self modifying code in a "fuck you" performance sandbox.

Right. Boosts set_fcontext even allows to omit the floating point registers (saves a lot of time if the application doesn't do fp)

This doesn't bother me as much. X86 could add lightweight instructions to clear large parts of the AVX state (a la VZEROALL) along with their associated "in use" flags, and compilers could be taught to emit these instructions whenever they're done using the big registers. Then context switches could mostly avoid touching them.

??? it's way more than that. at least 32kb if they keep # of vector registers same.

Oh for pete's sake, the die size cost for supporting the legacy x86 instructions is a fraction of a percent on a modern multicore CPU. This is because the translation hardware stays relatively fixed (in terms of size and complexity and transistor count), while Moore's Law keeps adding more hardware to each core and more cache, etc..

Take a look at this annotated die shot of an old Pentium 4:

https://i.stack.imgur.com/QK4gm.jpg

Up at the top is the microcode memory and microcode sequencer. That's the hardware that's responsible for translating all those old, large legacy instructions. It's not really much space on the P4, and the P4 Northwood is a 55-million transistor CPU.

Nowadays, depending on if you're talking about a mobile part or a higher-end desktop or server part, the CPUs have between ~1.5 billion to 2 billion transistors.

Again, that microcode ROM just doesn't grow very much as you add instructions, even a ton of instructions. And this is, well, one reason why Intel just keeps adding instructions. It's practically free.

People frequently complain about the complexity of decoding the x86 ISA and that seems to be what you're arguing against. But I didn't see the OP as making that argument at all. Instead it's things like the way x86 handles integer condition codes that has a huge effect on the complexity of the reorder buffer because now you have to track this additional input for each of your instructions.

But quite apart from the transistor cost of various legacy feature is the engineering cost. When designing a new x86 chip you have to make sure that every instruction works correctly in every mode that supports it and that's hard. NRE is a large fraction of the cost of each chip and verification is a big part of that. Intel had all the transistors in place to support SMT early on in the Pentium IV development but they didn't turn it on until Northwood because it was hard for them to make sure it worked reliably. And recently look at the problems Intel has been having getting TSX to work right or the problems AMD had with legacy modes in Ryzen.

And also, decode might not take very many transistors but those transistors are in constant use and consume way more power than the huge seas of 8T SRAM cells in the local cache despite being hugely outnumbered. This is far from the majority of the chip's power but every little bit counts.

You're right when it comes to die size and transistor budget, but complexity can sometimes have hidden costs. Complexity can limit and constrain the design of other aspects of a system in ways that are nearly impossible to predict and often non-obvious even when you're down in the trenches.

Not only can today's features limit tomorrow's design, but because those features give rise to mental models and habits of thinking they can limit future design in an almost unconscious way. The more complex your system becomes the more its existing complexity dominates your thinking to the exclusion of innovative ideas.

This process happens so quickly in software that it's obvious there:

http://www.retrologic.com/jargon/R/rococo.html

The whole impact of the architecture on the processor is to be considered, not just the size of the decoder. If the ISA was irrelevant (and the ISA does not just influence the decoder, it influences pretty much everything in various degrees) in modern days and even semi-modern days, Intel, with all its power (vastly superior to Arm, from a financing pov), would have come with something that does not suck in mobiles given how strategic this market is. It has not. In other words, you should not confuse the size of the lexicon with its associated comprehensive semantics.

It does not matter that adding instructions in that framework is not too costly. The framework itself is costly for some applications...

> Intel, with all its power (vastly superior to Arm, from a financing pov), would have come with something that does not suck in mobiles given how strategic this market is.

This market is a lot more price-sensitive than the desktop, laptop and server CPU market.

That's a great graphic! I see a few more from around 2003 on http://chip-architect.com/, do you know of anyone who's doing modern chips (x86 or ARM) this way?

It's not just ancient instructions that add costs, P4's MMX is also outdated etc.

Further the core issue is the cascade where A limited B and C is limited by B directly and A indirectly, now continue that though 15 generations and things get complicated.

Is 1 transistor always 1 transistor? Afaik most of the increased number of transistors and die size comes from larger caches. I could imagine the translation hardware having a much larger impact on cost, complexity and performance than yet another few million transistors of L3 cache.

Note: I have zero experience in hardware design

Hey John, your in depth articles on Ars are missed.

Ever thought of getting back on writing them occasionally?

I wish Intel (IACA, VTune, ...) or Agner Fog would list instructions which divert to the MS-ROM. A 2 or 3 instruction equivalent is often faster.

I think many commenters here are viewing this only in terms of die cost. But the blog post is not about that: it's about the other costs that this complexity introduces! Each new feature interacts with all the others in complex ways, and each new feature requires significant new engineering efforts by anyone who has to interact directly with x86 assembly (I sketched a few of these use cases below [1]).

The same author has also written [2] about Intel's CET (their attempt to add control flow integrity (CFI) support at the CPU level):

> The tail end of Intel's document is rather terrifying; it tries to enumerate the interactions of their CFI feature with all the various execution modes that Intel currently supports, and leaves me with the impression that they're generally heading over the complexity event horizon.

[1] https://news.ycombinator.com/item?id=14560267

[2] http://robert.ocallahan.org/2016/06/is-control-flow-integrit...

Surprised to see RISC-V is not mentioned. They take an interesting approach to extensibility where it's basically envisioned in the ISA from the start. Outside a core subset of features that programs can assume, you have to check each extension you need before you use it, so (in theory) it should be possible to drop deadweight extensions in future.

See "ISA Overview" here: https://content.riscv.org/wp-content/uploads/2016/06/riscv-s...

> Outside a core subset of features that programs can assume, you have to check each extension you need before you use it, so (in theory) it should be possible to drop deadweight extensions in future.

x86 processors also have cpuid

> https://en.wikipedia.org/wiki/CPUID

to signal which extensions are supported or not. In theory it would also be possible to drop deadweight extensions this way for x86, too. The practice shows that at least for the tasks that PCs are used for, one can hardly drop anything here. Exceptions are

- 3Dnow! (is not supported in Bulldozer, Bobcat and Zen anymore; cf. https://en.wikipedia.org/w/index.php?title=3DNow!&oldid=7696...)

- A20 gate (not supported anymore since Haswell: cf. https://en.wikipedia.org/w/index.php?title=A20_line&oldid=78...)

Agreed, but then again there just isn't any practical implementation available yet. If there was a raspberry pi like device, I think we would start seeing it mentioned more.

I'm guessing you mean workstation/server scale implementations? There is a shipping microcontroller implementation and has been for a while now.

We should have an RV64G implementation this year with any luck.

x86 has CPUID which is not all that different.

The problem with feature detection is that you can't depend on programs to be well-behaved. For example in Web applications you often see accidental or deliberate assumptions "if we detect X, then Y must also be supported". Often there is a !Y path that has never been tested and therefore doesn't work.

This is entirely anecdotal, but a coworker of mine worked at amd, and he once told me that amd did a study and found that instructions needed for backwards compatibility accounted for less than 1% of CPU die space. Big silicon manufacturers are pretty price sensitive and the decision to break backwards compatibility is a clear cut economic one. Clearly they have chosen not to break old software at the expense of continued architectural complexity and maintenance.

The problem isn't the hardware devoted to decoding or even executing instructions so much as the overall design and verification complexity. The existence of 286 call gates has implications throughout the memory subsystem and now when you're adding your TSX transactional memory instructions you have to think carefully about how they interact with call gates and whether you could be introducing bugs and make sure you have unit tests to uncover those bugs.

Yes, you're right. The silicon giants decided it was cheaper to fund that testing and verification work than to break backwards compatibility.

Except they occasionally miss some nasty bug in that area...

This is yet another danger to security.

At the risk of sounding like a broken record, I like what ARM did with AArch64. AArch64 is not entirely a clean break with 32-bit ARM, but it eliminated a lot of legacy mistakes and is generally a radical shift in the direction of a clean RISC design. Unlike x86-64, which piled 64-bit support on top of the existing ISA, keeping most of the legacy intact, AArch64 aggressively addressed problems such as conditional execution, PC-as-register, the small register file, LDM/STM complexity, barrel shifter complexity, etc. Most importantly, 32-bit mode is an optional feature not mandated by the ARM architecture, hopefully paving the way to dropping the legacy stuff entirely in the future.

ARM doesn't have to support legacy binaries in the same way that the wintel ecosystem does. I don't think it's really comparable.

Is it really a problem to emulate this in software, though (honest question)?

As far as I can see, if compability with old software is the only concern, then losing 80% performance isn't really a problem since the new CPU is probably 5x faster anyway. Thus making up for the loss, so the old executable runs as fast on a new CPU, using emulation, as it did running in hardware on the old one.

So all we'd really lose is a speedup for legacy software -- and who really needs to play Wolfenstein 3D at 80,000 FPS, as long as it works?

Yes, it is a problem. The new CPU won't be 400% faster, but maybe 10-20%. And initially there won't be any new ISA software, so all software will use the old ISA. Thus the new CPU will be much slower for "all" software (from the users perspective) because of the need for emulation. This causes the new CPU to be a bad deal for users, thus causing developers not to release software with the new ISA.

This is probably an important reason why IA-64 failed, but x86-64 succeeded, Itanium CPU's ran x86 software at about half the speed for a much higher price.

The ARM solution seems like the least suboptimal solution in this case, as it at least eventually allows support to be dropped, when emulation is no longer a problem.

The only large-scale successful transition of an ISA migration without backwards compatibility is when Mac Os X transitioned from PPC to x86. It probably helped there that the Apple ecosystem is a bit special and much more closed and controlled, and that the Intel Core Duo's of the time had much better performance than the IBM G5's.

Relative to x86, a lot more ARM use cases are in the embedded space, where one vendor writes all the software that will ever run on the CPU. So I think there's merit to the idea that it's easier for ARM to break backwards compatibility than Intel can.

Of course, a lot of ARM use cases also don't have that luxury. Android apps using the NDK are the most obvious example. The Raspberry Pi is another example: the RPi3 is still on 32-bit, held there by binary blob drivers and the massive amount of effort needed to create the fully open source driver stack that's a prerequisite for moving over (Eric Anholt is doing an incredible job; it just takes time).

Do you remember how Apple Macs moved from Motorola 680x0 to PowerPC? And then to Intel?

They had fat binaries and slow emulation as a stopgap solution. But these become mostly unused pretty soon.

That is, there is a possibility to move to a new ISA if most of the software will be recompiled with minor changes, and there's plenty of will to upgrade. This could work for server software.

Not so with Windows desktop software, of course.

On the server side, one can use quite good Power8 servers right now, or ARM-based servers, depending on the workload. An Intel CPU with a non-x64 architecture might have a good chance against them. But the transition would be hard if Itanium or Transmeta are any indication.

Itanium also did it by slapping on another die that was an x86 cpu. The core Itanium instruction set where totally new and different at the time.

If the new thing were and x86_64 CPU with just the oldest, least used and least useful features removed, things that haven't been used intentionally by new software in the past 15 years the problems you describe could largely be avoided. The gains would not be as big as cutting out the all worst features, but there would be some gains for minimal pain.

Potentially this could be paired with a deprecation process that Intel, AMD and other x86 (Are there any other) manufacturers agree on in an open setting with ISVs. If this working group announces the deprecation of some features and the ISVs agree its a good idea, then the chip makers could put out a date some years in the future where that feature would no longer be in new CPUs.

Sure they do: people want to keep playing Candy Crush when they buy a new phone.

Candy Crush is compiled to more or less architecture-independent bytecode on Android and a dual (32/64bit) binary on iOS. So no worries wrt Candy Crush.

There's a lot more to CPUs than the size of their registers. The fact that those 32/64 bit binaries still run is exactly the kind of legacy support we're talking about.

iOS 11 coming out this fall won't support 32-bit apps.

It is interesting that Intel finds it in this position. Their extend-extend-extend mantra (briefly interrupted in the Itanium until AMD pushed them back into it :-) has left them with a lot of "junk DNA" in their chip. That isn't horrible, the transistors are small and spreading them out helps avoid hot spots, but as pointed out it's also a waste.

But this is perhaps the first time in history that there is a realistic chance of them being replaced by an alternate ISA. The largest markets for "computers" (and by that I mean the ones that people compile on and develop systems on and run data centers on, have largely adopted open source (if begrudgingly) and that means you can get functionally identical software (but not performance identical) on different ISAs.

So at that point what keeps Intel alive is their volumes make it so much more cost effective than a bespoke architecture. And then there is ARM which is 'open' in the sense that anyone can get a processor architecture license, and 'high volume' as the basis for all of the compute appliances.

But that effort has been slow going[1] and continues to falter on corporate models that want to jump to the level of penetration today x86 has in a heartbeat rather than through measured evolution.

[1] https://www.nextplatform.com/2017/02/01/arm-server-chips-cha...

Wasn't there some dead weight dropoff† once with amd64, as a x86 CPU is unable to execute 16-bit code when in 64-bit mode?

IIRC that was a notable cause of DOSBox becoming slower when going 64-bit as it had to emulate the 16-bit CPU instead of just running on metal, as well as 64-bit Windows (7?) being unable to run old DOS and Win3x software.

With an increasingly rolling-release culture on software I can't wait for X-year support for hardware instruction sets /s. Wait, this is already happening for Android phones, where ARM SoC manufacturers just flip it at OS and phone vendors (Nexus 4, OnePlus X). I just realised Apple made a damn right call getting that under their control on that front.

† the metal may still be there though, waiting to be jettisoned as the market for 32-bit OSes goes away.

In long mode you no longer have access to the virtual 8086 mode which is most likely what DOSBox was utilizing.

> as a x86 CPU is unable to execute 16-bit code when in 64-bit mode?

This is still possible, though unsupported by Windows and Linux (the latter one never supported 16 bit code).

32 bit x86 Linux supported 16 bit code.

http://man7.org/linux/man-pages/man2/vm86.2.html

The vm86 syscall is for entering Virtual 8086 mode, which is indeed not supported in Long mode.

This mode was for example used under Windows 9x for implementing the DOS box.

But this is not what I was talking about. I am talking about 16 bit code in protected mode. This is/was used under Win16 and is supported in all 32 bit versions of Windows.

Win16 can run in 64bit mode as long as it is 16bit protected mode code. WINE uses that to run Win3.1 apps in 64bit Linux. For example:

http://i.imgur.com/KtgLeWu.png

(this is an old shot of mine and yes i used a XFCE theme to make it look like Win95 :-P)

> Win16 can run in 64bit mode as long as it is 16bit protected mode code.

Not under Windows: https://news.ycombinator.com/item?id=14246521

DOSBox was always an emulator, at some point they added JIT but it was always emulating the CPU, it never run code on the underlying CPU directly.

The silicon is still there, but it's not going away. Intel loves their backwards compatibility. It's also marketing: if Intel dropped support for some instructions, AMD could say they support more instructions than Intel even if those instructions aren't used.

When in "long mode", meaning a 64-bit OS, you can execute everything other than 16-bit protected mode code. You can still run 16-bit code that was designed for real mode. This means you could run some really old DOS programs, but not ones with a 16-bit DOS extender and not 16-bit Windows programs.

Of course, a 64-bit OS could pop in and out of 64-bit mode as needed.

Yep they threw out a bunch of stuff. What I remember: the CS, SS, DS, and ES segment registers, binary coded decimal instructions, protection ring 1 and 2 as only 0 and 3 are used.

Probably there is a lot of more things that were removed.

> protection ring 1 and 2 as only 0 and 3 are used

Are they really dropped in x86-64? I thought they are still supported also in 64 bit mode.

>> waiting to be jettisoned as the market for 32-bit OSes goes away.

I didn't know there still was a market for 32-bit x86 OSes. I suppose legacy systems - SCADA and such.

There's a market for 16-bit OSed too, though often they run virtualized because the hardware might not even exist anymore.

Interesting proposal for a forward compatible instruction set architecture by Agner Fog:

http://www.forwardcom.info/

This will no doubt raise hackles, but instruction set architectures are FAR less relevant than they used to be.

Fewer and fewer systems or people care about x86 vs ARM vs RISC-V. C has been the intermediate language of choice for decades now, and things like WebAssembly drive that care even lower. And if you really have to, you can binary translate.

x86's real problem is Intel's business model.

To compete today, you need processor IP you can integrate with other components on your die or system -- that's why ARM wins outside of the data center. In theory, Intel could sell x86 designs a la ARM, but that would completely break their business.

My bet: x86 gets squeezed between the rise of GPU vector supercomputing and ARM general purpose CPUs. Apple is already doing this in-house.

"Fewer people care about the ISA" is one of Intel's "real problems". Intel having a near-monopoly on the x86 architecture is a lot less valuable than it used to be.

On the other hand, while it's true that languages and runtimes provide more portability these days, there's still important handwritten assembly, and the amount of investment in compiler implementations still gives popular ISAs an edge.

I mostly agree, but I think history and expectations play into it as well.

I can imagine customers that claim they don't care, then some ISV releases some software or game that only runs on one platform, then all of sudden the customer cares. The ISV might not be malicious or in cahoots with anyone, perhaps they just have 2 decades of experience releasing for wintel and haven't reconsidered their approach.

In a world where JIT become more and more common, I could see many legacy features being dropped at one point.

Alternatively, I wonder how feasible it would be, for a multi-core system, to keep one legacy core for incompatible processes and faster new-arch cores for running the main beef.

The current generation of new programming languages are all compiled to machine code (Rust, Go, Swift)

That's an implementation detail. Code written in these languages is generally trivially retargetable.

You could also use some form of pre-compiled-non-platform specific system, which you then compile/optimize to the current platform.

Ideally we should also be able to decompile and recompile code. But this might be hard for x86 die the complexity.

The post posits that, to handle the burden of legacy instructions, an "obvious technically-appealing approach (is) starting over with a clean-sheet architecture".

The approach that immediately occurred to me would be have a layer that translates the legacy instructions into modern equivalents; without as much concern if they are slower to execute in their new form (they're legacy, after all, right?).

Of course, doing something like that is probably nowhere near trivial, the devil's always in the details.

But I bet this is already being done at the microcode level. Stepping things up to having a published agreement about which instructions were globally considered "legacy", and guidelines for what their equivalents were, would go a long was towards allowing a general feeling that an ISA was evolving, rather than just accumulating weight upon weight.

> The approach that immediately occurred to me would be have a layer that translates the legacy instructions into modern equivalents; without as much concern if they are slower to execute in their new form (they're legacy, after all, right?).

This is what x86 CPUs already do and have been doing for years.

It's also why Intel & AMD don't care about "x86 complexity" and it's also why even though people love to claim that a switch to RISC would improve efficiency/performance, so far there's really no evidence to support that.

Intel just plops their teeny-tiny (in terms of die space) x86-to-internal-microcode transactor on top of their new cores and calls it a day. As long as that translation layer isn't a bottleneck, which it rarely is, then Intel doesn't care.

It's not just about legacy instructions which can be decoded down to some microcode. It's about architectural features like SGX, CET, MPX, TSX, VT --- plus the legacy stuff like segment registers and 286 call gates and virtual 8086 mode and so on and so on --- and how they all interact with each other, and how they increase the complexity of context switching, OS support, and so on.

OS complexity is because they choose to keep and value that backwards compatibility, just like x86 values backwards compatibility.

Linux could eliminate all that complexity tomorrow by just bulk removing x86 support and only running x86-64. All OS complexity eliminated. All context switching complexity eliminated.

> legacy compatibility features [...] those CPUs won't be as efficient, cheap, reliable or secure as they otherwise could be [...] we can't rely on increasing transistor density to bail us out.

Legacy features shouldn't have a large impact on transistor usage, and they should have almost no impact on efficiency (although that may involve design choices). Almost all of the transistors are going to be used to implement the cache, routing between the high level blocks (external IO, cores, shared cache), and routing between the various microarchitecture units.

Using Broadwell as an example, the CPU cores[1] only use a relatively small amount of amount of area. However, even that is misleading, because most of the acre within each core is dedicated to the microarchitecture, which implements a far smaller set of micro-ops. Inside each core[2], the only place affected by legacy instructions is the decode stage (mostly in the "4-Way Decode" block in [2], probably less in the "PreDecode" block"), and a larger MicoCode ROM.

The legacy instructions might not be as efficient as they used to be in older CPUs; as low-priority instructions they may be implemented with less-convenient micro-ops that were designed for other, faster instructions. The software that uses those instructions was probably designed for a far slower CPU anyway, so it's probably no important if those instructions are a "slow" when the CPU overall is much faster.

I agree that the interactions between instructions might increase the difficulty of security, concurrency, or other higher-level concerns, but that will depend on the specific instruction. I'm not sure how large these concerns might be, unfortunately, so I cannot say if it's a serious issue.

> so their complexity ends up being "dead weight"

This sounds suspiciously like the unfortunate recent practice of labeling "old" things as "bad" or "holding back progress" and thus must be removed. In most cases backwards compatibility is a very important feature that adds very little cost. The low cost is possible because you can usually implement older features in terms of the newer features. This is true in CPU microcode, just like it's true in software.

[1] http://i59.tinypic.com/2mrxso9.jpg

[2] https://en.wikichip.org/w/images/thumb/a/a1/broadwell_block_...

Would like to hear what someone with compiler writing experience has to say about this. Is it really that much of a problem?

I'm not sure compiler writers care that much; they don't have to support every x86 instruction and can restrict themselves to whatever subset produces fast code.

Where it really kills you are things that have to analyze x86 binary code (e.g., binary static or dynamic analysis tools). Unless you cover every nook and cranny of x86, chances are you'll eventually encounter some program that uses one of these instructions and your analysis will break. And of course emulators like QEMU face the same problem.

This problem becomes especially severe in the context of malware – any instruction you forget to model becomes a way for a malicious program to evade your analysis.

I can't really answer that but your question reminded me of https://www.strchr.com/x86_machine_code_statistics

Turns out that in practice a tiny number of instructions make up the vast majority of all used upcodes in a typical application (although it's not clear in this article how long the tail is, there could be hundreds of instruction in that "others" slice).

My understanding is that there are many legacy instructions in x86 that are basically never used nowadays. They're still a problem for silicon designers because they still have to implement them for compatibility purposes but compiler developers can chose not to use them and instead select a more "risc-y" subset of instructions to do the bulk of the work.

It is not just the complexity but also the question of economics.

In general Intel on the server side is under pressure by specialized chips. Incremental spend will rather go into specialized silicon, SSDs and RAM than into CPU where it delivers more bang for the buck.

On the PC side people are looking for battery life - something Intel lags. Both Apple and Microsoft are eying the ARM universe causing Intel to recently fire a patent threat in the direction of Redmond and Qualcom.

From a volume perspective Intel is big but ARM with both mobile and IoT behind it has the momentum.

The tick-tock has slowed. New generations are becoming harder to manufacture.

Moore's law ending will assert itself in economic terms. Complexity is not your friend when the volume is increasing elsewhere and adding value becomes harder.

AFAICT, infrequently used instructions are implemented with microcode. I would guess that each of these instructions have a cost of a few thousand transistors, on a chip with billions.

But aren't there thousands of those instructions?

Well there is a rumours that Intel is working on a cleaner and leaner x86.

http://wccftech.com/intel-developing-new-x86-uarch-succeed-c...

The fun thing is that if they got rid of things like segmentation (make segment register instructions fault), 90%+ of modern x86 user mode code would still work.

The biggest threat to x86 is containers. A new unit of work removes the lock in. Then also things like Android and ChromeOS in addition.

The other aspect is cloud providers have the data for the next generation of processors.

Containers must be native to the host architecture.

Reply on news.ycombinator.com