That said (subjectivity ahead), the mouse becomes annoying only when I'm done thinking, and it's time for doing. Imagine you've got your window manager set up just so, terminal multiplexer sessions all pointed in the right places; and finally, a problem -- and clear solution -- in mind. All that's left is implementation.
You start typing your solution, switching windows to compile or refresh a webpage every now and then. Things are coming together. As you proceed this way, a stroke of brilliance hits you, and you add an extra flourish that resolves an entire class of problems. Welcome to the "Flow State."
Finally, you realize you'll be needing a file from your Teamviewer session with another computer. You foreground that process, and navigate to the remote computer's file manager. So far, you haven't even had to traverse farther than three keys from the homerow.
Unfortunately, to transfer a file over Teamviewer, you'll need to use the mouse. You'll have to click three times to get to the file sharing widget, and use the mouse even more to get your remote/local directories lined up, another click elsewhere to initiate the transfer... close the little confirmation popup that's now obscuring your screen. It's not the end of the world, of course, but your dance has been interrupted.
I'm not sure you'll relate to anything I've stated above, and I'm by no means advocating "going mouseless" as objectively superior to your own preferred flow. Whatever works for you, works for you. I am hoping that my reply might help you get in the head of a keyboard jockey.
By the way, this reply was written with a rather fluffy cat sitting on my mousepad. So I'm not the only one to be inconvenienced when I have to switch to a mouse. Needless to say, George is a huge fan of vim. :)