Hacker News Reader

Comment by ajross | Hacker News Reader

I'm not sure I understand your example. The Alphas had a pretty standard MESI cache architecture and a very typical cache coherence behavior. It's true that individual CPUs require barriers to enforce ordering, which is not true on x86 (where the architecture guarantees consistent ordering or memory operations between all entities on the bus, and spends significant die area to do that). But honestly x86 is the odd architecture here -- almost no one else felt that was a good tradeoff. (In hindsight, x86 was right, of course.)

Where Alpha was weird was in its floating point implementation (two versions on chip! to support the crazy VAX formats that no one cared about even at the time) and its alignment requirements (the hardware, not really atypically, couldn't make a misaligned load, but actually "could" by trapping into a microcode handler that was 10x slower, leading to mysterious insane performance regressions).

View on HN

The example isn't explained well in general, IMHO.

Suppose thread 1 executes the following code:

  x = 1;
  store_fence();
  p = &x;

At a hardware level, it's guaranteed that the cache coherency traffic to update the value of p is going to happen after the traffic to update the value of x. So it's natural to assume that means that anyone who sees that p == &x must have to see that x == 1 as a result of this traffic. And for most architectures you'd be correct.

But you would be wrong on Alpha. Alpha has two cache banks, and there is no coordination between them (in the absence of memory barriers). So if p and x reside in different cache banks, it's possible for a thread to load the value of p (observing that it is &x), and then load the value of *p and fail to see the assignment of x = 1--if the cache bank that contains x is somewhat overloaded on processing the bus traffic, for example.

Incidentally, you can't actually take advantage of the opportunity to avoid the memory barrier in this scenario on everyone-but-Alpha in the C++11 memory model, because it turns out that figuring out how to specify the (hardware) data dependency matters for ordering purposes in a source language is a lot trickier than it might appear. memory_order_consume (added for this purpose) is lowered to memory_order_acquire in all compilers I'm aware of.

I've seen it mentioned in several memory model texts that, while ARM's model is more "relaxed" than x86, the DEC Alpha's model is even more relaxed. For example: [1] says "The venerable DEC Alpha is everybody’s favorite example of a weakly-ordered processor. There’s really no mainstream processor with weaker ordering", and [2] says "Some CPUs (such as i386 or x86_64) are more constrained than others (such as powerpc or frv), and so the most relaxed case (namely DEC Alpha) must be assumed outside of arch-specific code." [1] adds, specifically, "[data dependency ordering means that] if you write A->B in C/C++, you are always guaranteed to load a value of B which is at least as new as the value of A. The Alpha doesn’t guarantee that."

[1] https://preshing.com/20120930/weak-vs-strong-memory-models/

[2] https://www.kernel.org/doc/Documentation/memory-barriers.txt

The Alpha did have a memory model that was weaker in some respects than pretty much every other RISC CPU. I'm not clear on the details, but I think that one of them is described in https://www.kernel.org/doc/Documentation/memory-barriers.txt in the 'GUARANTEES' section -- if you do "Q = LOAD P; D = LOAD *Q" then on Alpha and only on Alpha extra barriers are required to ensure that the second memory access (whose address depends on the result of the first load) really does get issued in-order after the first.

Why should the hardware do something software can take care of just as well was the design philosophy here.

Because when it doesn't, it leads to "insane" results like the one upthread. Reasoning about memory barriers is outrageously hard, and asymptotic software quality is outrageously expensive. At the end of the day an x86 kernel is going to have fewer race conditions than one for a traditional architecture, and that's worth something.

This is a lesson that software folks have been pushing with tools like static typing and memory lifetime analysis. The same applies at the hardware level.

You "just" need a memory barrier on every ptr indirection (including array indexing, etc.) that might go through something that's being shared among multiple threads. The more interesting question is whether requiring that additional barrier buys more performance in the typical case where you're not touching shared data.

> The more interesting question is whether requiring that additional barrier buys more performance in the typical case where you're not touching shared data.

Probably yes, i.e., there was some hardware reason for it. You can find one explanation [1] floating about that explains the effect was due to the cache banking design, with separate invalidation queues for each bank. Presumably that was better in some respect than the alternative designs available to implementors.

That said, the difference is probably fairly small and if Alpha were around today they would almost certain want to abandon that particular reordering since they are the only one doing it and there would be a lot of pressure on them to make it fast (this comes up in all sorts of non-trivial scenarios like double checked locking, reads of final fields in Java, etc).

That's generally the pattern, I think: if you are the odd man out with the weakest model, and you aren't the dominant player, you will have a lot of pressure to at least offer fast ways of lining up with the stronger models. So even if a weak model is a better design point in a vacuum, it might not be true when real-world implementation pressures are considered.

---

[1] http://lse.sourceforge.net/locking/wmbdd.html

Reply on news.ycombinator.com