I'm more talking about common things. Take the humble load, give a memory address, get some data into a register, what could be simpler?
Well for one thing your load may come in multiple sizes, can target different kinds of memory (e.g. device memory, non-cacheable memory, fully-cached memory, ARM architecture actually allows you to get quite specific about differing levels or shareability and cacheability too), can be unaligned with respect to the access size (but you still need full performance with them). There are ordering requirements with respect to other loads and stores in the system (even within the same CPU avoiding read-after-read ordering issues may not be simple) and various different kinds of barriers that can effect loads. You get exclusive loads or atomics (some variations can return the data seen so are performing a load on top the atomic op). It may be a vector load that needs to quickly feed the vector register file as opposed to the 'standard' register file. In a multi-processor system you can have various different types of snoop operation coming in that could effect the load. You also need to work out if you're actually allowed to do that load, modern page tables are pretty complex affairs. The page table itself could be changing as the load is executing. A decent chunk of the complexity is for virtualisation support but even without that there's various fiddly bits.
You also have speculative execution attacks to worry about. Certain loads may need to be very sure they should execute before doing anything, others may be free to speculate away and forward data into further speculative execution.
You can certainly build a perfectly functional ISA that avoids a lot of this (and avoid other things by keeping to a simple in-order microarchitecture) but that will loose you a lot of performance.