I can definitely recommend the Dvorak layout. If you think Colemak will be even better then go for it!
I need to be on holiday to learn the new layout though.
But personally I haven't found them a problem at all, even in Emacs.
I think now (2 years or so later) it's between 90-120 depending on how many typos I make.
You aren't really going to be super fast with dvorak but I do notice that my hands don't feel so tweaked.
If you learn a new keyboard layout, does that make it hard to use a regular layout? And what about if someone else uses your computer?
When you relearn the old layout, you can avoid your bad habits.