It doesn't have to be most people, with the traffic levels that Google probably deal with, even a small percentage doing that could add up to a lot of 'lost' click revenue.