This is a great point - when I was trying to articulate (to myself) why I prefer the keyboard it wasn't necessarily any speed advantage, it was more like the "flow" advantage. Your point speaks to that.

I think there's also a big difference in the amount of physical effort needed when using the mouse. With the keyboard it's easier to keep most of my arms relaxed, and when I need to switch to the mouse everything from the shoulder down has to contract into action.