I have the opposite feeling, because I implemented authorship on my blog and for that authorship picture to happen the URLs have to have high ranking, otherwise they won't get shown. That's why I have come to associate authorship with popularity.

On my blog only the 6 most popular articles have authorship info in the search results, generating about 300 visits from organic search per day. And my blog is a highly technical blog that targets developers and I haven't seen any drops in traffic due to authorship.

Also, people here make the assumption that regular folks behave like us. They don't.