We probably will never know what exactly went wrong, but I think the current best guess is this: Intel set minimum metal pitch target for its 10 nm process at 36 nm, while TSMC set minimum metal pitch target for its 7 nm process (equivalent to Intel 10 nm process) at 40 nm. These processes were the last before EUV, pushing DUV to its limit. It turned out that you can do 40 nm but not 36 nm. That's it.
It was an extremely technical issue, with justifications on both sides, and Intel's decision wasn't obviously mistaken, even in retrospect. 36 nm was usual scaling to Moore's law. It was 40 nm that was unusual, TSMC underscaled metal relative to fin and gate. Why 40 nm? Because it was the limit of DUV double patterning. But Intel also knew that! That's why Intel went with quadruple patterning. It was a conservative choice to underscale metal to the limit of double patterning, but quadruple patterning wasn't crazy either. Everyone was using quadruple patterning for fin at this point, so it wasn't thought that risky.
Should Intel have reconsidered this once they encountered yield issues? Yes, but that's hindsight. All processes experience yield issues. Intel probably thought they can be solved, faster than going back to drawing board and doing everything again. Unfortunately they couldn't.
(Above analysis was informed by lots of reading on this topic I did over the years, but https://semiwiki.com/semiconductor-manufacturers/intel/7433-... in particular.)