Ugly thought - if "almost all" accesses are Read-Write consistent, maybe an exception list of addresses could be used?

Basically do the 4byte cell approach, and make most of your RAM the "fast" endianness, with some 4byte cells "correct" endianness, because you know they are being accessed in a way that cares. (Since you're interpreting, I'm guessing this is reasonably-easily detectable?)

It does put a "if-addr-on-exception-list" on your mem access path, but if that's a hash lookup it could be OK. If you can further guarantee that your text segments are always fast-endian, then your opcode dispatch loop can still go direct and miss out that test.