Basically everyone learns to type without looking at some point which is the ideal zone you want users in (a one-to-one connection with the machine, eyes straight, hands down).
Context switching the UI in a computer interface, often randomly depending on the app, while requiring the user to look down to use it is terrible UX. This whole technology screams ‘because we can’ (aka there was space to do it) rather than developers needed help with anything.
The worst part was including Escape key into it, which is where most of the complaints come from as it’s a critical part of modern OS and browser interfaces - not just vim/emacs.
By including the escape key it went from a voluntary / side interface for certain use-cases (complex music, photo, or editing apps for example) to becoming part of the primary keyboard/HCI itself which is where it crossed the line from a mediocre idea into a bad idea.