I think most of the time it's because everyone has experienced getting low quality results from SEO optimized sites. It's getting to the point where one cannot shop or search for product recommendations on popular search engines because you'll get nothing but toptenreallyspecificproduct.com or absolutebestproduct.com.
Optimizing for search rankings doesn't say anything directly about any given site, but it's generally a signal for low quality content that would not rank highly without such optimizations.