Surprising new feature in AMD Ryzen 3000
agner.org
agner.org
[1] Basically the same as register renaming, but instead of using the register file to rename architectural registers, it can rename memory instead.
Even at that point something similar was described in Wikichip (which I quote at the above link).
Long before that, going back to Zen 1 I saw talk of the memfile, although I never saw a microbenchmark measuring the effect on on Zen 1: it seems much harder or impossible to trigger there (on Zen 2 it is very obvious: any basic store-then-forward loop using the same addressing expression will show a latency of 1).
---
[1] https://gist.github.com/travisdowns/bc9af3a0aaca5399824cf182...
Interesting that the forwarding is claimed to be done under speculation. I wonder if it is really necessary. Also, is it value speculation or does it checks the state of the cacheline?
Edit: thinking about it more, this is probably just the existing alias predictor that only cares about adresses.
The optimization has to happen early, in the front-end (around allocation) at which point you'll not know if the load actually aliases the earlier store [1]. To get the zero latency, you take a guess and correct yourself if you are wrong.
The non-speculative approach is basically the existing pre-memory-renaming approach: you probe the store buffer looking for earlier stores to the same address. This has also gotten much faster (3 cycles total on current Intel), but that's still a lot slower than 0 cycles (or 1 cycle total on Zen 2).
> Also, is it value speculation or does it checks the state of the cacheline?
I assume memory ordering is handled in the usual way, with an ordering nuke if a concurrency related misordering is detected. About how it's verified, I think it could either verify that the load actually came from the corresponding store (e.g., do the load and check that the SB entry used was the same as the predicted one), or it could verify that the value was the same: i.e., do the load and check that the value was the one used earlier based on the prediction.
The latter would succeed in some additional cases. It should be easy to test the two cases (essentially, Agner's add test with different register, but add 0 not 1).
---
[1] As a special case, you could prove that some loads alias some earlier stores when the addressing expression is the same (or can be proven equivalent), and there is no intermediate store, but evidently the AMD approach goes a ways beyond this since it allows for intermediate stores.
Yes, this is what I meant by alias predictor. It is probably a variation of the exiating aliasing predictor that allows reads to be speculatively reordered before previous stores of unknow address, except that on this case you would rollback on address mismatch instead of match.
That said, I also thought they might feed this info back to the front-end in order to implement this.
However, Agner's testing seems to show otherwise: it seems that "does it alias" is determined basically by a symbolic comparison of the addressing expressions. That is, [rax] matches [rax]. OTOH, [rax] doesn't match [rcx] even if those registers hold the same value. Also [rax + rcx * 2] matches [rax + rcx * 4] even though the scale factor is different! Perhaps it was too much work to compare the scale factors.
The example 20.3 where he incremented the value in between the load and the store using a different addressing expression (but the same address) and it consistently took a misprediction seems to indicate that they are not using an aliasing predictor in the usual sense since this always-fails case should be easy to predict.
Overall, quite interesting and some surprises there.
So except for a small window in the middle you have the same pressure on the PRF.
I don't know if that is how it is actually implemented, of course!
But I must admit, the initial impression of a 'forum' where all the topics are from one person looks a little bit like it might be a crazy person debating with themselves. (In this particular case, it looks like people other than Agner also create topics). Ideally you'd maybe re-skin the forum?
You can inline images and code easily. Easy to self-host, so you can own all the data, and you get solid support for comments so you can avoid data-peddling companies like Disqus.
I wouldn't go as far. Forum software is notoriously annoying to configure securely and to maintain.
> you get solid support for comments
... at the price of people having to create Yet Another User/Password Set, and likely bother you at some point for user-admin tasks.
You definitely keep control of the data though, and some people really like forums.
I'm not familiar with setting up forum servers specifically, are they difficult to setup for just one user? I can totally see how having proper authentication, emails, caching, etc getting complicated, I'm just curious if it's any user for personal use? Obviously you lose the ability to receive comments though.
Setting a forum up for one user seems like a big waste of time, when there are plenty of perfectly useable blog engines out there that are simpler and more secure to run. Unless, of course, one is already an expert in a particular forum engine.
It’s certainly an interesting way to do it. I’m not sure id set up a forum just to write notes though, there’s far simpler methods. I think in this case theres already a community on this forum and it’s easier to just write there than anywhere else.
https://docs.rs/dtolnay/0.0.9/dtolnay/macro._01__await_a_min...
1. lots of reasons, probably.
The author also has a standard HTML page for content that doesnt require comments: https://www.agner.org/optimize/
https://mspaprophet.tumblr.com/post/166690894965/the-mspa-pr...
Using latencies from Zen 1 instruction table (see https://www.agner.org/optimize/instruction_tables.pdf):
mov dword [rsi], eax ; MOV m,r latency is 4
add dword [rsi], 5 ; ADD m,i latency is 6
mov ebx, dword [rsi] ; MOV r,m latency is 4Each instruction depends on the result of the previous, so we need to sum all the latency figures to get the total cycle count. Is this right? How does Agner make it add up to 15?
Then for Zen 2:
mov dword [rsi], eax ; MOV m,r latency is 0 (rather than 4,
; because it is mirrored)
add dword [rsi], 5 ; ADD m,i cannot find an entry for this.
; Looks like there's a typo in the doc.
; I guess the latency is 1.
mov ebx, dword [rsi] ; MOV r,m latency is 0Again, how does Agner make it add up to 2?
And for Intel Skylake:
mov dword [rsi], eax ; MOV m,r latency is 2
add dword [rsi], 5 ; ADD m,i - latency is 5
mov ebx, dword [rsi] ; MOV r,m latency is 2 functional unit renamer
add tmp_reg, eax, 5
st [rsi], tmp_reg mov ebx, temp_reg
BTW, I'm hand waving on the micro-ops and FUs. There's probably some address generation, ... going on that I'm leaving out. I don't even think the renamer requires a translated micro-op. You could replace tmp_reg with renamed(ebx).
Now the downside. What happens at an exception? You have to back all of this optimization out.
Just what always happens at an exception. You just replay everything up until the exception occurred.
"If anybody has access to the new Chinese Zhaoxin processor, I would very much like to test it."
Will be very interesting to see how much actual changes Zhaoxin made to the VIA cores. I'd expect it to be minimum.
I'm curious if this change is an effect of more transistors (more space for a bigger register file) or if they're taking advantage with the microcode translation of the fact that most code doesn't use the SIMD vector registers and re-use unused parts of the register file for these memory aliases.
This is different from previous approaches where in-flight register data was held in the ROB and later written back into architectural registers.
If you're doing random access to memory like that, you're probably out of the realm of what is appropriate in vector code and should be looking at other hardware (c.f. a GPU's texture units) to manage your memory access.
But the scatter/gather instructions do random access memory operations. You have one SIMD register with a 8 (or whatever the width is) indexes to be applied to a base address in a scalar register, and the hardware then goes and does 8 separate memory operations on your behalf, packing the results into a SIMD register at the end.
That has to hit the cache 8 times in the general case. It's extremely expensive as a single instruction, though faster than running scalar code to do the same thing.
Now I was surprised by Agners figures for zen2 LOOP and CALL which both have reciprocal throughput of 2. Being equal to doing with just normal jump instructions.
Skylake on the other hand has 5 or 6 for LOOP and two CALL variants with 3 and one variant with 2.
Edit: The answer from LLVM's source code is yeah, gather loads are microcoded on Zen.
I don't understand how do you think it could change or break a volatile or lock-free algorithm?
It's a transparent optimisation - it doesn't change observable behaviour. Just like how register renaming, top-of-stack-caching, and so on and so on don't change observable behaviour.
The only difference it would make is timing.
I can't write such a program. Can you?
How would you ensure that another thread was scheduled between these instructions in order to observe some intermediate state? If I just told you the OS would never schedule between these instructions again is that breaking any contract or something you can observe somehow? How would you tell?
If we can't such a program, then the logic isn't needed.
In one thread, do a loop of:
loop:
movq [rsi], 10
addq [rsi], 5
cmpq [rsi], 15
jeq loop
In another thread with the same value in rsi, do:
loop:
movq [rsi], 100
jmp loop
Would it terminate on this and chip? How would the distribution be different?
The store to the address at rsi will actually go in to the processor's store buffer and the subsequent add is allowed to retrieve the value from the store buffer _before_ it's drained to main memory.
Under TSO, AMD's memory mirroring implementation here is totally fine.
It could potentially change observable behaviour if you could turn it on or off but this is irrelevant given you wil see changes in observable behaviour for that same snippet of code depending on which microarchitecture of AMD or Intel processor you choose to run it on.
It's worth having a read on the details for different memory models and how they relate to implements. This seems like a good guide: https://www.cs.utexas.edu/~bornholt/post/memory-models.html
Why would you expect it to eventually terminate? There is nothing in the architecture that should lead you to expect that.
The architecture does not guarantee that instructions from the first loop will be interleaved with the second in any particular way, no matter what cores they're running on.
> The probability distribution of the time it takes to terminate would probably be informative.
But again... the architecture does not guarantee the set of or probability of distribution of interleavings. You cannot write a sound program using this information.
while (flag1 == true && turn == 1) { /* wait */ }
If the variables were mirrored to registers, and that mirroring was not invalidated by another thread's write, then the loop would never terminate.
Of course Zen2 did not break Peterson locks, so the GP's question is cogent and has the answer "yes, there is such logic."
It's just a guess, but that's probably what happens in order for the memory mirroring not to break the cache-coherency. So "yes, there is such logic", but it already existed in the processor to avoid the same problem when data is in the cache. And so, there would be no additional logic to handle this.
Notice that your code has a backward jump between the write and the read and the example in the article (and the code of the other person I replied to) does not.
There is no logic needed to interrupt the renaming such as a signal from another core holding the cache, the rename just naturally retires beyond a certain point.
Read Morris Marden's 2014 patent on it for more details.
https://patentimages.storage.googleapis.com/16/3f/47/ea08705...
The patent is a great resource, thanks for that. But I think it supports my point when it suggests that memory renaming may persist indefinitely:
Alternatively, reclamation of the original physical register could be delayed to facilitate later memory renaming.
and later:
...the MRT may always retain physical registers of N past store operations...the physical register is reclaimed to the free list when it exits the MRT, for example, when it is no longer one of the N most recent store operations.
So it's possible. I don't think we know from Agner's work whether AMD went this route or not.
This optimization has a latency of 0-1 cycles.
If you write memory first to address A and then to B. Another core never can see the B change without also seeing the A change at any moment in time.
But in this case there is only a single memory location. So there is no ordering that could be violated in the first place.
It's like the ABA problem - how do you differentiate between just never happening to see a B out of chance, and B being elided instead? You can't.
As far as volatile: this isn't changing anything to do with memory operations. It's just caching a computed address in a special place so the CPU doesn't have to decode and recompute it for the very next instruction.
I think volatile in general is pretty useless, and it'd be better to replace it with atomic style functions:
x = volatile_read(&v);
y = use(x);
volatile_write(&y);
If you're paying attention to the standards community, what I'm wishing for was mostly sketched out in ISO/IEC TR 18037, section 6. (http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1169.pdf)
People try to use it for thread coordination. It's always been wrong.
Yes, he is wrong.
He has a point, though.
https://en.cppreference.com/w/cpp/language/cv
> Every access (read or write operation, member function call, etc.) made through a glvalue expression of volatile-qualified type is treated as a visible side-effect for the purposes of optimization (that is, within a single thread of execution, volatile accesses cannot be optimized out or reordered with another visible side effect that is sequenced-before or sequenced-after the volatile access. This makes volatile objects suitable for communication with a signal handler, but not with another thread of execution, see std::memory_order). Any attempt to refer to a volatile object through a non-volatile glvalue (e.g. through a reference or pointer to non-volatile type) results in undefined behavior.
In fact, Standards notwithstanding, real compilers routinely optimize away volatile operations if they deduce nothing can see the object -- e.g., it is on the stack, and its address has not been taken. And they will happily re-order reads from and stores to other objects around a volatile operation. Furthermore, caches don't know squat about volatile, so they will eagerly re-order the volatile ops, besides.
So, no, he did not have a point. He was just wrong, wrong, wrong. But it never caused him any personal distress.
But that is what he insisted.
"When the CPU recognizes that the address [rsi] is the same in all three instructions..."
Is there another abstraction layer like some CPU code that runs that would do the "recognition" or is this "recognition" happening as a result of logic gates connected in a certain static way?
To put more broadly: I'm really interested in understanding where the rubber meets the road. What "code" or "language" is being run directly on the hardware logic encoded as connections of transistors?
Note that Intel chips have many (at least 10) execution units, and AMD also has 10. You need to split instructions to the different pipelines for efficient execution. I forget exactly how much... but far more than one pipeline in any case.
If pipeline0 is busy doing a divide, run additions in Pipeline5. Tomasulos algorithm converts typical assembly language into this multi-pipeline form. It also keeps track of which pipelines are busy, and finally puts things back together in the right order when done. (Division can take 80 clock cycles, while addition takes 1. You gotta stall the results of the addition instruction until after division is complete. If division came first in the machine code)
I guess it's unsurprising to find that even at the single cpu core level, there's another layer of parallelism. And that will of course complicate things further.
When it comes to actual execution of the final microop itself, its surprisingly simple. You put the bits in the right location, then a circuit magically calculates the result.
XOR and Add are simple enough (https://en.wikipedia.org/wiki/Carry-save_adder). A multiplier is harder to think about, but yes, its just a bunch of wires hooked up to NAND gates: https://www.slideshare.net/suryakrishna3785/wallace-tree-mul...
Division and Modulo are traditionally executed with microops in a multi-step process. Addition becomes subtraction through the magic of 2s complement. AND, OR, XOR, are trivial logic gates. Memory lookup is complicated, but mostly due to caching. The core concept of memory lookup (put value onto address pins, wait for memory to respond with the stored value) is simple enough.
Memory is traditionally on a "bus". The idea of a "bus" (multiple different CPU components talking on the same set of wires) is hard to grasp. But when you learn about tri-state logic (https://en.wikipedia.org/wiki/Three-state_logic), the 5V state, 0V state, and "High Impedance" state, it becomes simple.
"High Impedance" means that someone connected to the wires simply won't read, or write, on those wires. The CURRENT (amps) going in, or out, is set to zero through the magic of transistors. If you have 5 different components (C0, C1, C2, C3, and C4) on a set of wires, if C1/C2/C3 are in "High Impedance", then C0 and C4 can talk without any issues (because C1 through C3 won't use up any electricity from the wires).
The key is coordinating who can talk and at what times. But that's where "link level protocols" start to come in. Tri-state logic (in particular: high impedance) is what makes it possible.
And there ya go. A fully working CPU. That's all it is.
Well... optimized addition isn't simple. But any beginner can implement naive carry-save adders on a breadboard (or Verilog/FPGAs) and get a functional Adder (maybe even ALU) in a weekend.
But you're right. To optimize addition (in particular: the propagation of carries), you need to use a Kogge-Stone Adder, which is pretty sophisticated (https://en.wikipedia.org/wiki/Kogge%E2%80%93Stone_adder). And that's from my Bachelor's degree: I'm sure there are newer algorithms that are more sophisticated than Kogge-Stone.
There's a lot of issues at play here at the electricity level. IE: Fanout. All components can only carry so much current (amps). For example, a transistor may only be able to feed 5 to 10 other transistors, so building "Buffers" to increase the current so that you can actually turn on all the transistors is a thing.
Each CMOS transistor has capacitance: so you need a certain amount of electrons before the transistor turns on. Given a level of current (amps), this means you need to wait for enough electrons to get onto the input before the transistor responds.
Doing all of this with the minimal energy usage and maximum speed (minimum latency) is surely complicated. But the "naive" version suitable for beginner play is pretty simple. Like raytracers: you can write a raytracer in just 100 lines of code (it won't be as fast as a professional raytracer, nor have as many features, but you'd get the gist in a single weekend project: https://github.com/matt77hias/smallpt)
---------
In any case, once you get to the final decoded instruction, its "just" a circuit. Sure, Kogge-Stone and Wallace Tree multipliers need a bit of study before you understand them. But they're simple black-boxes. Stick the bits into the correct wires and then a few hundred picoseconds later, you got the result coming out of another wire.
These (8 and 16) are still small numbers. I wonder how important renaming is on ARMv8 or RISC-V with 32 GPRs. I really wonder if compiler register allocators help/hurt/know about renaming which in fact would require microarchitectural knowledge.
The example that Agner found is a form of register renaming. So is this AMD trying to make its legacy code base go faster (AMD+Intel do a TON of that) or is it something a compiler can actually target? I think the former.
Of the 32 GPRs on ARM's Cortex A78 application processor, there are 160 renaming registers.
> I really wonder if compiler register allocators help/hurt/know about renaming which in fact would require microarchitectural knowledge.
The benefit of register renaming is that idioms like "xor eax, eax" automatically scale to whatever the reorder register size is.
"xor eax, eax" REALLY means " 'malloc' a register and call it EAX". Because the reorder buffer changes between architectures and even within an architecture (smaller chips may have smaller reorder buffers), its best to "cut all dependencies" at the compiler level, and then simply emit code where the CPU allocates registers in whatever optimal way.
The compiler doesn't care if you have 160-renaming registers (ARM A78), 224-renaming registers (Intel Skylake), or 300+ registers (Intel Icelake). The "xor eax, eax" idiom on every dependency cut emits the optimal code for all CPUs.
----------
You should have a large enough architectural register set to perform the calculations you need... in practice, 16 to 32 registers seem to be enough.
With the "dependency cutting" paradigm (aka: "xor eax, eax" really means malloc-register), your compiler's code will scale to all future processors, no matter the size of the reorder buffer of the particular CPU it ends up running on.
EDIT: It should be noted that on Intel Skylake / AMD Zen, "xor eax, eax" is so well optimized its not even a micro-op. Literally zero uop execution time for that instruction.
https://stackoverflow.com/questions/11227809/why-is-processi...
There are older editions available free online that cover the same concepts, they just don't have the very latest info.
In short no microcode but there is a PLA (Programmable Logic Array) which helps to decode the instructions.
[1] http://www.righto.com/2016/12/die-photos-and-analysis-of_24....
On current-gen CPUs, only rarely used instructions are decoded into multiple micro-ops, the majority of them decode into a single micro-op, even when they do many things at once like memory load + compute. More often the opposite happens, multiple AMD64 instructions are merged into a single micro-op, this is called “micro-ops fusion”.
Proof: Intel's LEA instruction is executed as one uop. That's a super complicated instruction that you'd expect to be split into other uops, but... lo and behold, complex all the way down to the end silicon.
Even a "risc" machine: ARM has macro-op fusion, with AESE / AESMC instructions. (Macro-op fuse AESE / AESMC into a singular AES macro-op. A full AES cycle includes both steps, so its very rare to ever need to only do one of the steps).
As such, I've considered "RISC" to be a misnomer. Even RISC-V and ARM are now doing macro-op fusion (and micro-ops). So what exactly is RISC mean anyway?
very minor nit: it is actually called 'macro-op fusion', 'micro-op fusion' is when ops from a single instruction are put back together in part of the pipeline.
Otherwise, spot on.
(This article is grist to the mill of my "every CPU eventually evolves to become an interpreter for its dominant programming language" thesis.)
[] It always struck me as one of those propeller-head features that GCC & co. love but which the MS and Intel compilers avoid 'just to be on the safe side' :]
But most people don't enable these features when building because they often make the resulting executable/library unusable on general CPUs. However, businesses/individuals with more specialized needs and better control over their targets often do capitalize on these.
-march=arch -mtune=arch
Interestingly, Intel's Clear Linux - a optimization-oriented distribution - uses
-march=westmere -mtune=haswell
The first I could think of (in the 5 minutes I've read this) is that it could be potentially ASLR breaking. As Agner says, it's highly useful for stack operations, but that also can help an adversary derive where the stack is.
Second, I could see an attack where the CPU predicts a sequence will use this register (since the address in the register could be speculative) and prefetch (since it kind of looks like it's prefetching) but it ends up being wrong and not killing/clearing the register, or not preventing its usage speculatively.
But it all depends heavily on lots of things. I feel like it's pretty similar to spec v4 in that exploitation would depend on the memory disambiguator, but we'll see.
This will of course not be able to tell you all the black magic that is happening inside AMD's and Intel's newest designs. The software optimization manuals for these processors do include some more interesting insights, but especially the Intel ones are not the most entertaining read...