Wait, so you use both colemak and the normal qwerty layout and switch between them on the same keyboard?

As a vim user, I am hugely relying on muscle memory - which is why saying that bindings should vary between layouts seems really weird, as that would make muscle memory way less effective. Then again, that is my usage ..