Hacker News Reader

Comment by SteveOllington | Hacker News Reader

lol, poor Matt, always under fire from everybody who hasn't got the rankings they want. It's simple really, Google try to make it fair for everybody (and granted it doesn't always work perfectly, which is why the continue to make changes, it's evolution) and if some sites give themselves an unfair advantage and are caught, then they're punished. How else could a search engine work? If it were any different then al of our results would be greed pages with no useful info and everybody would stop Googling stuff.

As somebody once quoted, "You can please some of the people, some of the time, but you can't please all of the people, all of the time!"

View on HN

This is a fair comment but I believe the issues stem more from the process Google follows.

It is obvious that Google cannot communicate exact reasons why a site was penalized as that would help spammers. However, there is nothing that prevents them from adding a step to warn the offending website and give them a heads up before the ban/penalty takes place, along with an explanation of the policy that is/was being violated.

Most of these heads up would go ignored, some would not and yes, it would incur a support cost. However, the number of websites which are significantly penalized isn't onerous... I believe fewer than 1,000 each year?

When a company has become the defacto gateway to the internet, I believe they have a responsibility to webmasters. Google has lost a lot of goodwill over the years because of these seemingly arbitrary penalties... Instituting such a practice would be a worthwhile investment.

option1138, I'm afraid you need to recalibrate your expectations of spam on the web. Blekko made a site called Spam Clock that estimates 1 million spam pages are created every hour: http://www.spamclock.com/ .

There's 200+ million websites out there. 1,000 spam sites would be a spam rate of 5.0 × 10^-6. If you remember the days of Altavista before Google, the actual rate of spam on the web is much higher. Here's one stat: I once heard a search engine rep (not from Google) say that they had to crawl 20 billion pages to find 1 billion non-spam pages.

So yes, we do tackle more than 1,000 websites a year. There's a ton of spam on the web, and Google has to operate on the scale of the web (e.g. in 40 different languages) to tackle all that spam.

Matt,

You are of course correct. The fault is mine for miscommunicating... I find myself becoming less self-editorial these days when I write on the web and tend to think everyone is on the same page as I am.

I was actually referring to an informal study I did earlier this year. I measured sites which were receiving an average of 50,000 or more visitors from Google US search (organic) per month over a six month period. Then I compared those with a similar set from a subsequent six month period to see which had significantly dropped off in traffic and rankings. The purpose of this was to estimate the number of significant sites which were penalized over that period of time. The final estimate came to about 700 sites/year which were penalized. There are lots of uncontrolled variables here of course... but I was looking for an "order of magnitude" answer simply for curiosity's sake.

The 1 million spam pages created per day were of course excluded from consideration as they never received much traffic from Google in the first place.

So just to clarify my earlier response, I am advocating for a policy that would apply to websites exceeding a certain threshold of organic traffic for a significant period of time.

lol, poor Matt, always under fire from everybody who hasn't got the rankings they want.

This isn't about desired ranking. If your highly useful, popular site gets blacklisted from appearing on Google at all, there better be a damned good reason.

Reply on news.ycombinator.com