Hacker News Reader

Comment by btown | Hacker News Reader

There's an interesting dilemma here - if the algorithm were to favor "pages with lots of content alongside the exact-match phrase you're looking for" then it would incentivize content farms that combine lots of StackOverflow answers together. And if you favor the opposite, where there's less content alongside the exact-match phrase, you incentivize content farms that strip away content from StackOverflow before displaying it. Ideally, of course, you'd rely on site-level reputation - is Google just having trouble recognizing that these are using stolen content?

View on HN

> is Google just having trouble recognizing that these are using stolen content?

It's very possible. In general Google will penalize you for duplicate content, but that might not apply to code snippets since code is often re-used within projects or between projects.

The code snippet sites also typically have 2 or more snippets on the same page. When combined, it might then look unique to Google since their algorithm probably can't understand code as well as it understands natural text. Just a guess

Also, while I imagine Google probably has (or at least easily could have) code-specific heuristics at play, it seems like it may be harder to reliably apply duplicate content penalties to source code listings, especially short code snippets.

Between the constrained syntax, literal keywords, standard APIs and common coding conventions it seems like even independently-authored source code listings will be superficially similar. Certainly the basic/naive markov-chain-style logic that works really well for detecting even small examples of plagiarism in natural language content isn't going to as effective on `public static void main(String[] args)` or whatever.

Obviously there are strategies that could distinguish superficial/boilerplate stuff from truly duplicated code, but between the volume of actual (and often legitimate) duplication on the internet and the (maybe?) low ROI for duplicate content penalties for code relative to other value metrics/signals, maybe this just isn't terribly important as a quality factor?

Wikipedia is a good example. Google considers it a high value website (less so than before but still) while it only has a single article about each topic. Other projects have entire websites dedicated to the single topic, products, experts on staff, live support. I presented as an example to some google engineers the wp page on suicide that explains popular ways of doing it vs dedicated prevention projects. Today (for me) it ranks the topic: 1-3) news articles, 4) world health organization statistics, 5) wikipedia, 6) Suicide prevention.

ABC news, the NYT, WP and the WHO are considered high profile but the topic is not their area of expertise. Non of them would consider themselves the go-to place for people searching for it.

Reply on news.ycombinator.com