That's a misconception. Yes, decoding x86 is somewhat more complicated, as far as I'm aware that's mostly because instruction length differs at a byte granularity. Still, the area dedicated to it simply isn't that large on those huge out of order designs.
I'm sure the instruction encoding plays some role, but I suspect what we're seeing is rather down to consistently good micro-architecture execution over the years and Apple being ahead of everybody else on manufacturing process.
Compared to the x86 processors that exist today, M1 also benefits from having the memory in-package.
Intel tried to be brave with Itanium, but failed because compilers weren't up to the job at the time.
Compilers "failed" because VLIW is fundamentally not a particularly useful idea. The main problem for general purpose single-thread performance is memory latency, and VLIW just doesn't help there.