Hacker News Reader

Comment by userbinator | Hacker News Reader

So there is logic to determine if a different thread has modified the value between those instructions?

Before you ask if there is logic to determine if another thread modified those values, first ask yourself if you can write a program where you can observe the difference if there was this logic or not.

I can't write such a program. Can you?

How would you ensure that another thread was scheduled between these instructions in order to observe some intermediate state? If I just told you the OS would never schedule between these instructions again is that breaking any contract or something you can observe somehow? How would you tell?

If we can't such a program, then the logic isn't needed.

If you have multiple cores then the OS’s scheduler doesn’t matter much.

In one thread, do a loop of:

  loop:
  movq [rsi], 10
  addq [rsi], 5
  cmpq [rsi], 15
  jeq loop

(Or however you write that with correct assembler).

In another thread with the same value in rsi, do:

  loop:
  movq [rsi], 100
  jmp loop

Ensure these threads run on different cores. You would expect the loop in the first thread to eventually terminate. The probability distribution of the time it takes to terminate would probably be informative.

Would it terminate on this and chip? How would the distribution be different?

If you are relying on that behaviour then your code is already broken on most modern x86 processors due to their memory ordering implementations.

The store to the address at rsi will actually go in to the processor's store buffer and the subsequent add is allowed to retrieve the value from the store buffer _before_ it's drained to main memory.

Under TSO, AMD's memory mirroring implementation here is totally fine.

It could potentially change observable behaviour if you could turn it on or off but this is irrelevant given you wil see changes in observable behaviour for that same snippet of code depending on which microarchitecture of AMD or Intel processor you choose to run it on.

It's worth having a read on the details for different memory models and how they relate to implements. This seems like a good guide: https://www.cs.utexas.edu/~bornholt/post/memory-models.html

> You would expect the loop in the first thread to eventually terminate.

Why would you expect it to eventually terminate? There is nothing in the architecture that should lead you to expect that.

The architecture does not guarantee that instructions from the first loop will be interleaved with the second in any particular way, no matter what cores they're running on.

> The probability distribution of the time it takes to terminate would probably be informative.

But again... the architecture does not guarantee the set of or probability of distribution of interleavings. You cannot write a sound program using this information.

You need some form of barrier (lock/mfence/cpuid) prior to compare instruction to make the snippet correct.

That's a very useful way to approach the question. Thanks for reminding me of that technique!

It's easy to write that program. For example, a Peterson lock, which has a busy-wait loop:

    while (flag1 == true && turn == 1) { /* wait */ }

the loop terminates by another thread writing to either variable.

If the variables were mirrored to registers, and that mirroring was not invalidated by another thread's write, then the loop would never terminate.

Of course Zen2 did not break Peterson locks, so the GP's question is cogent and has the answer "yes, there is such logic."

But the read of turn will not be eternally mirrored to the original write - at some point the rename file eventually moves on when the original store (an observable side effect) retires like a register renaming.

Notice that your code has a backward jump between the write and the read and the example in the article (and the code of the other person I replied to) does not.

There is no logic needed to interrupt the renaming such as a signal from another core holding the cache, the rename just naturally retires beyond a certain point.

Read Morris Marden's 2014 patent on it for more details.

https://patentimages.storage.googleapis.com/16/3f/47/ea08705...

Hmm, what's the advantage of doing memory renaming for unretired stores? Won't the value just be forwarded from the store buffer?

The patent is a great resource, thanks for that. But I think it supports my point when it suggests that memory renaming may persist indefinitely:

Alternatively, reclamation of the original physical register could be delayed to facilitate later memory renaming.

and later:

...the MRT may always retain physical registers of N past store operations...the physical register is reclaimed to the free list when it exits the MRT, for example, when it is no longer one of the N most recent store operations.

So it's possible. I don't think we know from Agner's work whether AMD went this route or not.

Store forwarding is slower, it is usually as fast a reading from L1 directly in the best case (4-5 cycles).

This optimization has a latency of 0-1 cycles.

In order for memory mirroring to be enabled, the memory must probably be read first since its value must be used. The processor must then ask the cache controller to acquire the cache-line in a "no writer / multiple reader" mode ("S" state). If another processor wants to write to this cache-line it must first take the cache-line in "single writer / no reader" mode ("M" state) which invalidates the cache-line in all other caches. In the processor performing the memory mirroring, if the cache-line goes out of the "S" state then the processor cannot read the cache-line anymore and memory mirroring has to stop since it reads this memory. In order to continue it must ask for the cache-line again, read it again, and then get an updated value.

It's just a guess, but that's probably what happens in order for the memory mirroring not to break the cache-coherency. So "yes, there is such logic", but it already existed in the processor to avoid the same problem when data is in the cache. And so, there would be no additional logic to handle this.

No - there doesn't have to be. As long as there are no barriers or other instructions which restrict memory ordering relative to particular instructions in the dynamic instruction stream, it doesn't matter what the actual interleaving is between different cores as long as there is a given execution order which is valid for the observable side effects.

X64 has stronger guarantees than that without barriers doesn't it?

x86 guarants that memory writes to different locations are seen in order by other cores.

If you write memory first to address A and then to B. Another core never can see the B change without also seeing the A change at any moment in time.

But in this case there is only a single memory location. So there is no ordering that could be violated in the first place.

Yes, quite a bit stronger.

In guarantees the order of visibility of writes to shared locations. It dose not guarantee the granularity of updates to the shared locations, which is the essential difference here. How do you observe the difference between the three instructions just naturally happening to all run before your next instruction completes, and them eliding the memory operation as is happening in this optimisation?

It's like the ABA problem - how do you differentiate between just never happening to see a B out of chance, and B being elided instead? You can't.

#StoreLoad still requires a barrier even on x86.

The cache-coherency must already ensure that. Memory mirroring conditions should include the permission to read (or write, depending on usage) the data cache-line.

Reply on news.ycombinator.com