Vi is incredibly well-suited and ergonomic for text editing, and switching between the modes is efficient, since often times you want to begin typing text as a consequence of some other normal mode operation you'd want to take anyway - e.g. add line below; change until end of line; change in parens; change up to character - and those operations put you immediately into insert mode afterwards anyway.
So I wouldn't let that feature (which enables making common operations more ergonomic) stop you from using vi-style binds.
Though modelessness can be quite nice, and I do see the appeal. Being able to select a region and immediately perform any desired operation is convenient.
What's great is that Emacs is what you make it - you can mix and match the two approaches. I, for example, use vi binds for text editing in Emacs, but maintain modeless binds I can always access when it makes sense. You can have the best of both worlds :)
> What keeps me from using vi derivatives
That's correct. There's no such thing as "vim-mode". Every single Vim extension is full of glaring inconsistencies and shortcomings. Not a single VSCode extension, not IdeaVim for IntelliJ, not Sublime ones — none of them are really full-fledged vim integrations. Emacs one is the notable exception. Evil mode in fact is so good, it never feels like an afterthought, it feels like it was baked into the design of the editor from the get-go. It works extremely well. Emacs in fact can vim better than (Neo)vim.