I can currently type 110-125 WPM on QWERTY, which I've projected would take upwards of a year to re-achieve on Colemak, which is a total no-brainer if I could reasonably expect positive long-term gains.
However, from what I've read (and heard from friends on Colemak), switching back to QWERTY is painful, so my typing speed on anyone else's computer (which I realized, after trying to switch, I do fairly frequently) would go down to 30 at best.
The primary argument for Colemak is that it reduces finger travel distance from the home row. But my fingers don't naturally fall precisely in a line: my left middle finger naturally drifts between the space of E and D, my middle and ring finger on my right hand drift up closer to I and O.
Culturally, being on Colemak means the world is suddenly against you. It's like choosing to be a left-handed person in a right-handed world: vim keybindings don't make sense, applications and flash pages ignore your custom keyboard layout and switch you back to QWERTY, and every single keyboard|laptop|smartphone you buy for the rest of your life will require a certain amount of tweaking before it's maximally usable (think iPhone autocorrect).
Keyboard layout hacking in general, however, is fantastic. Here's a few tweaks I've accumulated that have immediate benefits for little of the above downsides:
1. remap `caps lock` to `ESC`: goes without saying for vim, but extremely useful in other situations too (like playing starcraft and de-selecting units)
2. remap 1234567890 to !@#$%^&*(): as a programmer, I use these symbols far more often than I use the numbers. Switching to shift to numbers with symbols as default makes all the sense in the world. It no longer takes finger gymnastics to surround something in () or negate a term with !.
3. (optional) shift doubletap for caps lock: I missed caps lock after a while, so I brought it back with a double-tap of shift.