I completely disagree with this sentiment. Everything in a UI that you can do with a mouse should be tolerant of disabilities that people who use a mouse might be going throug (tremors etc). A user shouldn't have to try to do something and become frustrated in order to discover that it is "not for people with poor motor skills". We should do better. What you describe as "well worthwhile" still strikes me as woefully incomplete and annoying. Every user could potentially have poor motor skills at some point due to temporary causes.

It's not the end of the world of course. I just wouldn't think of "I have used it before and it was just fine for me" as a good reason to not strive for a more universal approach to interface design.