Hacker News Reader

Comment by Matt_Cutts | Hacker News Reader

I've been tracking how often this happens over the last month. It's gotten much, much better, and one additional algorithmic change coming soon should help even more.

I'm not saying that a clone will never be listed above SO, but it definitely happens less often compared to a several weeks ago.

View on HN

My experience is the exact opposite: I am seeing many, many more clone sites in my search results in the last few months. It feels like it increases when I accidentally click a clone site.

This happens for more than StackOverflow clones. Mailing lists, Linux man-pages, FAQs, published Linux articles, etc. all have clone pages that are obvious link farms (sometimes they even include ads that attempt to harm my computer) that rank higher than the "official" (or at least less-noisey) pages.

Ideally, I'd like to completely remove domains from result as has been discussed elsewhere on HN. Hopefully this upcoming push for social networking that Google has will reintroduce a better-implemented "SearchWiki" feature...

Try DuckDuckGo. Gabriel has been doing an aggressive job about removing unsavory domains and I've been fairly impressed. I think that Google probably can't be nearly as aggressive for political reasons.

The reason Google isn't doing the same thing as DuckDuckGo is most likely because manually banning a domain instead of improving their algorithms to avoid unwanted behaviors will only temporarily work, and only in select cases. There will always be new spam and content farm sites.

There seem to be a small number of large content farms (perhaps suggesting economies of scale are pretty important). In this case, manually killing them will work well for Herr. Weinberg.

Over at blekko, we leave in a few large but marginal sites like eHow, and let users kill them with their personal spam slashtags. For smaller spam websites, we can frequently use Adsense IDs to kill them in groups.

I think there is a disconnect in the scale at which you are both commenting:

Comment #1:

  > I've been tracking how often this happens over the last month.
  > <snip>
  > it definitely happens less often compared to a several weeks
  > ago.

Comment #2:

  > I am seeing many, many more clone sites in my search
  > results in the last few months

You can't argue against "things have gotten better in the last week" with "things have been getting worse for the last 6 months."

One specific and ubiquitous example of webspam has been driving me nuts this week: An enterprising spammer has won huge on Amazon Web Services (AWS)-related keywords.

For example: https://encrypted.google.com/search?hl=en&q=aws+s3+emr+p...

Result #4 at the moment is "AWS Developer Forums: Interactions between S3, EMR and HDFS ..." on http://www.hackzq8search.appspot.com/developer...com/...

What's sublime about this example is that:

1. hackzq8search is clone of AWS's websites amazonwebservices.com, aws.typepad.com, etc

2. hackzq8search is hosted on appspot.com, Google's App Engine domain

3. hackzq8search is over quota, so the site doesn't show any content anyway.

Yet this site was the top search result, beating out the site it was cloning, time and time again on my AWS/EMR-related searches this week.

The one mitigating aspect as that hackzq8search's URL naming scheme is easily decodable -- the hackzq8search URL includes the full URL of the cloned URL, so I can write a Greasemonkey script to extract the proper original URL.

I found a glimmer of optimism in that the site has been slowly fading in SEO-success this week: I complained about https://encrypted.google.com/search?q=aws+s3+security+sox+pc... on Thursday, but on Friday the hackzq8search Search Result was gone from the first search result page.

It's still not hard to slam some AWS-related keywords into Google and get these bogus results, though.

Someone else already reported this. There's been some weird stuff going on with AWS-type pages, e.g. see http://news.ycombinator.com/item?id=2103401 for example. I don't know the exact cause, but I know the indexing team is aware of this issue and working on it.

Is google ever going to go on the record about companies like demand media and whether or not they get special treatment from google?

While searching for a pdf file, I hardly find the pdf through the top results. The top results are often kind off pdf, ebook search engines themselves and they clearly appear on top by gaming Google. I hope this gets fixed too.

If you know it's a pdf that you're looking for, you can add filetype:pdf to your query.

This is great for HN readers. For most people using Google, not so much, I think.

Have you noticed sites that scrape google groups content ranking higher than the google group they've scraped? How can that possibly happen? Still seems rampant.

Thanks.

Why not just downrank the results from the SO clones?

Because that wouldn't solve the problem for clones of other sites, or clones in other languages. And the Stack Overflow cloners could just make other websites. That's why a primary instinct in search quality is to look for an algorithmic solution that goes to the root of the problem. That approach works across different languages, sites, and if someone makes new sites.

To be clear: the webspam team does reserve the right to take manual action to correct spam problems, and we do. That not only helps Google be responsive, it also improves our algorithms because we get use that data to train better algorithms. With Stack Overflow, I especially wanted to see Google tackle this instance with algorithms first.

But detecting duplicate content should not be very difficult, esp. now that Google indexes everything almost in real time. The site that had the content first is necessarily canonical and the others are the copies?

Because we don't understand what's hard, we think you're not really trying, and then we make up evil reasons to explain that.

I believe if people understood better the difficulties of spam fighting they would be more understanding.

> But detecting duplicate content should not be very difficult, esp. now that Google indexes everything almost in real time. The site that had the content first is necessarily canonical and the others are the copies?

Not necessarily. The rate at which Google refreshes its crawl of a site, and how deep it crawl, depend on how often a site updates and its PageRank numbers. If a scraper site updates more often and has higher PR than the sites it's scraping, Google will be more likely to find the content there than at its source. Identifying the scraper copy as canonical because it was encountered first would be wrong.

> Because that wouldn't solve the problem for clones of other sites, or clones in other languages.

Sites that are the victims of content cloning have to be very visible and valuable, so maybe a little manual curating could be relevant.

> the Stack Overflow cloners could just make other websites

Not really? The point is not to tag the clones but to tag the original; everything that is not the original and that has copied content is a clone -- its name, domain or country notwithstanding.

That would be a rather awesome feature for evil people: Just copy&paste your competitions content onto SO (or any other specially protected site) and their google ranking will drop like a stone.

Why doesn't Google just count 'original' as whoever published the corpus first...

this was discussed in an earlier thread and it seemed like the idea of finding the "original" gets really messy and game-able (and potentially oppressive if curation were used).

the primary input to search engines comes from web crawlers...the idea of "first" when it comes to duplicated content is already difficult to determine, and (I would guess) it would get much much worse in the inevitable arms race if something like this were implemented.

I agree with what you're saying: there needs to be an algorithmic approach.

But I'd like to say one other thing. Why is Google only doing something about web spam now after people have pointed out how bad things have been getting? Has anybody considered creating a small team to just oversee public perceptions of the search results and try to keep on-top of things in the future?

Do you think if Google educates to publishers and web masters to use Rel="Original" , the contents which are similar will be put it as considering set, use this tag as the best practice as similar to Rel Canonical tag that will help Google to identify the original content and make the search quality better?

What would stop the site that is scraping content from using the rel="original" tag too

Matt, would it make more sense to put more weight on number of web page visitors (recorded by Google Toolbar) as opposing to number of incoming links?

Reply on news.ycombinator.com