ARM Cortex-A72 fetch and branch processing
sandsoftwaresound.net
sandsoftwaresound.net
In a world where most code is JIT compiled on the target architecture and/or periodically re-compiled using the latest, optimized compiler once a new CPU comes out – sure.
But is that realistic? Maybe that would work for Android and iOS (with on-device and app store level compilation, respectively), but with Docker, we seem to be moving to the opposite direction again, and runtime hardware-based optimization makes a ton of sense there.
Why do and then un-do?
The OOO (= out-of-order) approach with high-quality prediction (e.g. branch, value) to processor micro-architecture makes sense if you have to mask a lot of memory-access latency, which comes from data-dependent (= unpredictable) memory access. General purpose workloads have a lot of that. (If memory access patterns are more predictable, you'd probably run your workload on a GPU or TPU or DSP, or some other accelerator.) Compilers, whether ahead-of-time, or JIT, have not got enough information to schedule commands in a way that can mask memory latency the way an OOO scheduler inside a processor can. Stalling the pipeline because you are waiting for data to arrive from memory is disastrous for performance and to be avoided at all cost. Intel's Itanium was based on the premise that it is possible statically to schedule well enough, but that has been considered a failure for general workloads. Moreover, if you let a compiler schedule you need to have ISA extensions that allow you to communicate scheduling information to the processor which is not cost free (for example you may spend precious bits to encode scheduling order that you then cannot use for other things).
I suspect that the very last word has not been spoken in this discussion, but everything obvious to make dataflow competitive (and quite a lot more) has been tried, and failed.
There is no conspiracy here.
If GPUs didn't come around when they did, and games were all CPU-based today, you wouldn't have an easy time inventing and selling a GPU.
> Intel itself tried to move beyond OoO and utterly failed.
One attempt in 2001 (and a few before then when the market was more competitive but before people had the context we have today) by the company with market power doesn't invalidate my point as much as reinforce it. People have tried approximately nothing in the last 20 years, all pointing back to this one bad design Intel did 20 years ago as if that proves the taken path is the only valid path.
I'm not convinced. You'd just need to show that hardware acceleration can produce worthwhile gains for computer graphics, which doesn't strike me as insurmountable. Perhaps start with an FPGA.
> People have tried approximately nothing in the last 20 years
Do you follow the 'Mill' architecture? https://en.wikipedia.org/wiki/Mill_architecture
Even realtime RT, which is mostly just a compatible add-on to raster, only managed to get off the ground when NVIDIA decided to go all-in and tank the criticism for a generation by effectively forcing it on the customer. No other company could have done that, even though Imagination was pushing for RT hardware for ages before NVIDIA.
> Do you follow the 'Mill' architecture?
Yes. https://medium.com/@veedrac/to-reinvent-the-processor-671139...
The problem is that building a CPU is very expensive and high risk, going even a bit beyond the classic OoO can bankrupt you. Intel tried with Itanic and AMD almost ate its lunch; AMD tried with buldozer and shared execution units and almost collapsed; SUN tried with flock of chickens and is no more; Nvidia tried with denver, and well, I guess they had a lot of cash. Both AMD and Apple just made completely orthodox designs and were complete successes.
Evolution is going to be in small steps, unless a new design can prove to be 10x better than the existing ones which is very unlikely.
I agree, I find it hard to see how you could abandon dynamic scheduling without unreasonable sacrifices.
My point isn't to say otherwise, just that we see less innovation than we would otherwise were our ecosystem built to handle it. GPUs have benefited hugely from the ability to rapidly change the ISA and a lot of internal details between each generation, and for each competitor to have their own take and extensions (Apple even has its own API, to support its TBDR features). If we had that sort of freedom in CPUs, I'm sure we'd have something a lot better than four variants of practically-the-same-thing by four different companies.
More likely: They recognized the failure and cut the program before losing too much money.
The NVIDIA CPU team is alive and well.
Unless in some alternate universe frequency scaling and memory bandwidth were never an issue, you're always going to come back to the reality that a parallel processor optimized for minimally-branchy math will be more efficient at graphics tasks. The early days of fixed-function hardware are, if anything, an aberration.
yes, because other workloads are niche and until very recently didn't warrant investing on dedicated architectures for them.
It reminds me of how MIPS branching is wasted on nops:
https://electronics.stackexchange.com/questions/28444/mips-p...
"The trick in writing efficient code is to put in an instruction that will be useful as part of the loop that is being taken executed, but do no harm if the branch is not taken.
The MIPS designers were counting on compiler writers to write clever enough code generators to handle this efficiently. However many do not (including Microchips C32 compiler, based on GCC), and just put NOP's after every branch, wasting both code space and cycles"
Spoiler: It didn't end well.
Had it not been the case and Itanium would have been pushed until it eventually replaced x86.
Without access to Intel patents who would come with enough money and ecosystem support to actually be a challenge to Itanium?
Intel, quite infamously, tried to make "something better" work with several generations of Itanium and failed impressively against the reality that a sufficiently smart compiler cannot be made to work. You can't blame any of the big players for holding back - Intel, MS, and HP were "all in" on Itanium as the 64-bit future - if it could have been made to work, it would have been widely adopted.
At the same time, Transmeta was building a X86-on-VLIW, which, guess what, also failed to be a competitive product. Nvidia, of course, decided to pick up from where Transmeta left off to build what ended up being an aarch64-on-VLIW design. In both cases, the hope was that the low-level translation would allow the benefits of a VLIW architecture while avoiding the "sufficiently smart compiler is impossible" problem, and in both cases performance just wasn't good enough to be competitive.
This really isn't the case IMO. Itanium is one attempt 20 years ago, it didn't try to rewrite the world (compatibility with standard C code is a huge crutch; GPUs couldn't exist in that environment for example), and the design sucked. And for 20 years since it has been sufficient to invoke the name to scare away any attempt of actually trying something new, even when they don't make the same mistakes, or even work anything like Itanium.
> Transmeta
It's in the name: the most significant part of the design was all the effort spent on compatibility with the dominant paradigm, running stock x86 code! That hugely limits what you can do.
> And for 20 years since it has been sufficient to invoke the name to scare away any attempt of actually trying something new, even when they don't make the same mistakes, or even work anything like Itanium.
People have been invoking Itanium for the last 20 years because that's what you do with expensive, painful lessons. Just like "branch delay slots are bad" (MIPS et al) or "no sub-word-size-access is bad" (Alpha) or "there's such a thing as too-relaxed memory ordering" (Alpha) or "putting too much magic in the ISA is bad" (iAPX 432), some lessons don't need to be repeatedly revisited and repeated.
I get that the Mill team wants to prove everyone wrong, but the only way to show that is working hardware and that doesn't exist (and given the time they've been working, probably won't ever).
> compatibility with standard C code is a huge crutch; GPUs couldn't exist in that environment for example
Except, guess what, Cuda and OpenCL look exactly like C in many ways. The major features they don't support (well) are dynamic memory allocations and branchy code - good luck trying to make an architecture work that can't efficiently handle malloc. People have tried a lot of different language+hardware architectures over the years, and all of them have been outperformed by OoO machines.
> compatibility with the dominant paradigm, running stock x86 code
If either Transmeta or Nvidia could have made a standout VLIW architecture, they would have had plenty of business. Defense and HPC customers have absolutely no problem with esoteric architectures so long as the performance is good enough. You could build the craziest architecture, and if you can show it'll be a winner, the DoE will buy it.
It was one ISA, and a terrible one at that, which is my point.
> Except, guess what, Cuda and OpenCL look exactly like C in many ways.
You're missing my point, which is about rewriting the world. C's syntax is not the problem.
Except, you know, the whole point of VLIW designs was that by offloading complexity to the compiler and runtime you'd be less limited by hardware design decisions and be able to improve with time. Provide real examples of how Itanium could have been done better.
> You're missing my point, which is about rewriting the world
No, I'm not. "Compatibility with standard C code" means memory allocations and branches. That's it. Most "real" OSs need an MMU, but there are a few research systems that don't.
You should also keep in mind that lots of different models have been tried. Compiled machine code? Done. JIT-recompiled machine code? Done. GC'd Lisp on hardware? Done. GC'd ada on hardware? Done. GC'd JVM on hardware? Done. Binary translation to "better" hardware? Done, lots of different times. "Rewrite the world" has been tried a fair number of times before, and it didn't win out.
You're also totally missing the point that several architectures, x86/x86_64 most notably, have succeeded because ISA and hardware implementation aren't locked together tightly. Processor designers and users are not nearly as constrained together as you believe.
It's really indicative of the problem I'm talking about that so many people think the options are either a standard, Intel Core-style OoO or Itanium-style VLIW. Not only is VLIW a massive range of potential things, and Itanium is not the natural implementation of it, but I never even said we should go with VLIWs! I'm personally more enamoured with CG-OoO style hierarchical reordering, though were rewriting the world possible I'd do something that exposed more of that to software (probably microthread style).
> Provide real examples of how Itanium could have been done better.
Although Itanium seems like a dead-end strategy to me, three things it got really wrong were:
* software data misprediction recovery,
* overly large register files,
* compiler target locks in uarch decisions.
(Please mind that it's been a while since I looked at the Itanium ISA.)
The Mill is sort of what Itanium would be with all the rough edges sanded over, though it's sort of a moot point since the Mill will never exist outside of talks.
Again, static schedules aren't the approach I'd advocate for, since there are some things they just can't do.
It didn't win because 1) making a top performing CPU is very expensive 2) you can only only recoup your investment by selling a lot of CPUs. So you need to be very efficient with the installed base of software..
While in the past this favored x86 / x86-64 compatibility, this is shifting towards ARM compatibility.. Maybe this will favour experimentations? ARM ISA is far less complex to implement so from a technical POV this could help companies trying to experiment.
That's only somewhat true. Until the modern era of cloud hyperscalers, fairly weird architectures with poor compatibility were quite successful, especially in HPC and defense markets. Cell and its derivatives were successful for their time. Any number of quirky microcontroller architectures were successful before the Cortex-M family came to dominate (and some still are). Yes, the startup costs are always going to be high, but success in the broader market isn't necessary for a performant architecture to survive.
> Maybe this will favour experimentations?
I don't think so, if anything the high performance aarch64 designs like Apple's M1 are more similar to x86_64 designs than previous generations. Large caches, wide execution, serious OoO, stronger-than-expected memory ordering. I think more experimentation will come from the RISC-V community targeting more specific applications.
And? If an 'exotic ISA' can really offer some improvement, it must be able to compete with 'the best of the best' implementation of current ISA..
Most software doesnt fully map to a data flow architecture, and when it does you can run it extremely efficiently on today's hardware anyway.
A system that AOTs and JITs and recompiles based on how the specific user actually uses the app and what it actually does on that specific device.
A new high level language that drops all pretense at imperativeness. Some combination of functional and reactive and other hip jargon where you lay down guidelines of what you need done with your assets, user inputs and events. No need to think about threads and shit. You no longer have to know what will actually happen on the metal. Let it go.
> A system that AOTs and JITs and recompiles based on how the specific user actually uses the app and what it actually does on that specific device.
Basically every Android device since version 7, and Windows Phone 8 execution model.
For example if an app tries to connect to the internet to download some news on every launch, but the user is often offline, the system magic would automatically inject a check for connectivity to avoid a lengthy timeout.
[citation needed] There are many difficult parts to a super scalar speculative dynamically scheduled microprocessor, but retirement doesn't even make top 10. If I had to pick, it would hands-down be the load-store unit which is really hard to scale (and has a lot of gnarly corner cases).
> This all may seem inefficient and crazy and it is. Program representation and execution need a major re-think along the lines of the once-investigated dataflow architecture.
I actually agree with the impedance mismatch of our legacy ISAs, but "dataflow" is a bizarre pick as the original formulation was incredibly inefficient and all modern processors (A72 included) are already dataflow machines over a fixed windows of the (partially speculated) path through the program.
There have been a lot of interesting paradigms published, but it's extremely rare for industry to take a risk. VLIW is still a darling (in principle very efficient iff you hit in the caches), but there are a _lot_ of other ideas. Browse through the past few decades of ISCA and MICRO proceedings to see examples.
The cost of making new binaries is always the problem. At Transmeta we used binary translation as the answer for x86 and NVIDIA's Denver much improved upon the idea for Arm. Both use in-order execution which in hindsight didn't seem to have worked out. Alas, dynamic translation of binary code has huge overheads and issues with cold code. There's hope though; if the code you are translating is WASM then you are in a much better place, but still not quite as good as source as you are lacking all the alias knowledge.
[updated typos and grammar]
There have been many attempts at making data-flow applications compatible with general purpose computation workloads. They all failed to beat the performance of comparable conventional processors. [1] lists the following disadvantages of data-flow:
- Debugging difficult (no precise state)
- Interrupt/exception handling is difficult (what is precise state semantics?)
- Implementing dynamic data structures difficult in pure data flow models
- Too much parallelism? (Parallelism control needed)
- High bookkeeping overhead (tag matching, data storage)
- Instruction cycle is inefficient
[1] https://course.ece.cmu.edu/~ece740/f13/lib/exe/fetch.php?med...