The issue was chipsets and peripherals, not POWER performance (which generally always beat x86 at the same point).
The problem was that the entire ecosystem was built around communicating with an x86. So, you couldn't get a Northbridge or Southbridge equivalent that was even remotely close in performance or power consumption to those in x86 space.
Unless you decided that you were going to take on everything in chip design, you couldn't compete. And Apple didn't decide to take on everything until Intel told Apple to pound sand and pissed Jobs off.