As a colemak user, definitely. It's a super weird and confusing experience otherwise and this bug is a blocker for me checking out your (otherwise quite promising-looking) project.
As a vim user, I am hugely relying on muscle memory - which is why saying that bindings should vary between layouts seems really weird, as that would make muscle memory way less effective. Then again, that is my usage ..
If the key on keyboard stays the same, even though the layout changes, it really messes with the way my brain maps from letters to muscle movements, idk if this makes sense.