I.MX7 M4 Atomic Cache Bug

This is dumb and "obvious" but the advantage of the layers of abstraction is that a lot of people share layers of abstraction. So the issues need to only be found once, so to speak

Of course you lose specialization and you end in "the image decoder supports some weird format which allows RPC " territory but most software can barely handle the high level bugs.

It's pretty crazy how something like Django exists, powers thousands and thousands of websites, but has perhaps less than a thousand contributors overall? Less than one person per project. Now that's an amazing payoff!

> how on Earth our technology is working as well it does, considering there are layers upon layers of abstractions which could have a lot of issues which are either worked around or just not hit in particular application.

Our technology works as well as it does because of these layers upon layers of abstraction. That's the only way you are going to be able to construct something with a few billion components and a fighting chance at avoiding unwanted interference between parts. The amazing thing is how often we get it just right, not that there are super rare edge cases that were not taken into account during the abstraction process that lead to bugs.

Every leaky abstraction is a bug in the waiting, all it takes is for someone to focus on the discrepancy with enough time, effort and resources thrown at it it might lead to a crash or an exploit.

Also note that it is not as if we don't know that caching is a hard problem to get right, it is one of the three explicitly mentioned in the 'there are two things hard about computing' joke.

I have exactly same experience.

And it is somewhat reinforced by the kind of "software problems" that only manifests themselves on new hardware revision or even production batch. And the causes range from wrong chip assembled in production (sot23-6, same package marking, but dual jfet really does not work as replacement for LDO), through various signal integrity issues ("how did you determine, that this makes 100R mictrostrip?" "It looked right and makes the board look right", somehow the "right-looking" diffpair on 6+ layer board does not work on 2 layer backplane) to real sillicon errata that either can (MSP430G-series USCI interrupt handling, you will not even trigger that when using TI provided sample code) or can not (Microchip's various pin routing issues, some Intel Atom erratas and such) be sanely worked around in software.

In that thread -- After reproducing the issue and performing some tests, it was found that the issue is because “LDREX” and “STREX” instructions overlooked LMEM cache. That means those instructions always access external memory directly, which leads to data inconsistency.

There’s no SW configuration to make the cacheable data consistent with those atomic instructions, and design team will fix it in later CM4 integration.

Its a bug. But they see a workaround so they aren't in a hurry to fix it apparently.

There's an even more similar bug in the PowerPC CPU used by the Wii U: all atomic operations have to perform a cache flush (dcbst) between the load-linked (lwarx) and store-conditional (stwcx) instructions, or else they won't work properly. (But I believe the issue is that the operations aren't being propagated from per-CPU caches to main memory, so it's sort of the opposite of the I.MX7 bug where operations are skipping the cache.)

I believe that the only generic fixes are to either (a) ensure DRAM is never cached, (b) ensure no atomic operations address DRAM (as the author proposed by using TCM), (c) implement atomics by disabling interrupts, performing a normal RMW operation (i.e. no LL/SC, as even with IRQs disabled that would cause cache incoherency), and re-enabling interrupts.

No library like FreeRTOS can guarantee (a). Not even the compiler can guarantee (a), since the user can control the cache by memory-mapped registers. (b) can't be guaranteed by a library, nor can it be guaranteed statically by a compiler, since only the linker knows where the atomic variables will reside in memory (and, atomic operations could be performed on an address that isn't a compile-time constant, e.g. dynamically allocated memory).

(c) also can't be guaranteed by any library, but it could be guaranteed by a compiler that has access to the full source of the binary. That's a hefty limitation though, since it means you can't mix any other compiler/language (e.g. assembly, which is almost always used for the startup sequence) into the binary and still have these guarantees.

For method (c), I believe gcc allows one to somehow override __atomic_load & other "builtins" - the use-case being that atomics can be implemented for new or uncommon architectures without modifying the compiler itself. If this is the case, then a potential fix could be shipped by a library (e.g. FreeRTOS) which defines __atomic_load as something like

  void __atomic_load (type *ptr, type *ret, int memorder) {  
  #ifdef IMX7_<partnumber>  
  __disable_irq();  
  *ret = *ptr;  
  __enable_irq();  
  #else  
  // insert code to perform a normal atomic load.  
  #endif
  }

In fact, gcc allows one to "wrap" functions - it might be possible to do something like

  void __wrap___atomic_load (type *ptr, type *ret, int memorder) {
  #ifdef IMX7_<partnumber>
  __disable_irq();
  *ret = *ptr;
  __enable_irq();
  #else
  __real___atomic_load(ptr, ret, memorder);
  #endif
  }

which would have the benefit that the library doesn't need to know how to implement atomic ops on other platforms. This approach could also be used to implement (b) by performing a runtime assert that `ptr` lives in a cache-coherent section of memory.

But again, this approach only works if you're not relying on any binary blobs that perform atomic ops. In the end, if you're doing anything nontrivial (e.g. atomic ops on heap-allocated memory, or even stack-allocated memory), it's impossible for a dev kit to completely hide this bug from the developer.

Alternatively, do these M4 processors have some type of updatable microcode like X86 processors do? NXP might be able to push a fix that somehow patches LL/SC primitives, or traps when they're encountered at runtime and allows the user to decide how to handle them (e.g. putting them in a no-IRQ critical section like above, but since it's done at runtime now you can mix multiple languages / binary blobs, etc).

> and never really dug into x86 so I'm not very familiar with the corresponding instructions there.

The x86 has no corresponding instructions (other than perhaps the very recent and not yet popular transactional extensions). Instead, the x86 world uses an "atomic compare and exchange" instruction. Wikipedia articles: https://en.wikipedia.org/wiki/Compare-and-swap versus https://en.wikipedia.org/wiki/Load-link/store-conditional