I think you're right. It's more about scanning the page and your "reflexes" than the actual speed of your clicking. I would say, though, that once you found the number you want to click on, the _accuracy_ of your clicking is important.