Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Google doesn't have a good way to establish provenance, and has trouble distinguishing copies from originals. It's a common complaint of blog operators that some bigger blog copied their stuff and got a higher ranking on Google.

Google could check when it saw something, but that won't work against fast scrapers. For that, you need trusted timestamps.

One solution to this would be to have a few time-stamping services. You send in a string, probably a hash, and it adds a timestamp, signs it, and sends back a signed result. Then provide a WordPress plug-in to use this service, hashing and time-stamping each blog entry, and putting the result in the HTML in some standard way. (Perhaps <span signed-provenance-timestamp-hash="xxxxx"> blog entry </span>). A few mutually mistrustful services for that would help; blogs with serious forgery problems could use multiple time-stamping services.

Search engines then need to look at timestamps as a rating indicator. If two results are very similar, the earliest one wins.

Isn't part of the core of pagerank that certain domains are more trustworthy than others?

Given that this is supposedly true, wouldn't the same content on stackoverflow out rank a random new domain with the same content

Exactly. In the SEO world, the common wisdom is you will get dinged in ranking or even deindexed if you publish duplicate content. Period. There is an entire sub-industry in SEO that's all about content spinning. I'm always kinda amazed that these sites show up and I usually get them for about 50%+ of my tech searches.

What I think ultimately happens is Google is notified and these sites are eventually deindexed. But this still leads me to really question the whole duplicate content theory sometimes. If Google was so tight about this, these sites would never show up in the first place.

It could also be that the tech terms searched for are not as competitive as, say, medical or diet or celeb terms. Maybe Google's just grabbing at any kind of relevancy at this point, duplicate or not.

This is really surprising. Given that they have so much data, there should at least be several thousand domains that they treat with some special regard when content that replicates them shows up.

Then again, I'm sure that there are plenty of top 1000 domains that would use this to their advantage too, oh well.

pagerank works both at the domain and page level, originally with the emphasis on page level(hence pagerank not domainrank).

While it would be hard for these sites to match the cumulative reach of SO, they will have an easier time getting specific pages to rank highly. This can also be abused with a system where most pages link to the .1% of pages that should be emphasized. In this fashion, smaller sites can throw their weight around and get specific pages to rank more highly than bigger sites.

Trying to solve this issue on a purely technical level turns out to be a lot more complicated than it would seem. It is made much worse by how damaging false-positives are for a search engine (Google censored me!). So the result is that this often only gets resolved by manual actions rather than automatic penalties.

Actually it's pagerank for Larry Page

hence his name is not Larry Domain.

Actually it's not. (In case anyone might think you were being serious.)

Since establishing provenance is such a big problem for Google, perhaps it might be a good idea for Google to offer a time-stamping service itself?

It's not directly a problem for Google — it's primarily a problem for sites that create original content.

Not quite. The quality as a search engine could improve dramatically by letting true authors let Google know the content is coming before it's been available anywhere on the web.

Well, I think Google would prefer to send the traffic to StackOverflow instead of it's clone as it's better source?

It's a problem that would disappear if content creators were compensated regardless of serving point.

We don't have that. We could.

Universal content syndication + broadband tax.

https://www.reddit.com/r/dredmorbius/comments/1uotb3/a_modes...

Exactly! It seems to me Google could easily provide a proof-of-authorship API, especially for text. (You could publish a hash in some feed in case you don't fully trust Google in turn.) I'm not a big fan of conspiracies and such, but by now I'm pretty sure Google has some conflict of interest given that it hasn't offered such a service already.

Problem as others pointed out is that no matter how you design the system, even with Google a completely as trusted party in this, proof-of-authorship, timestamping-service, it can only work if every original content creator (or platform, like SO) gets on this train.

Say you're a content creator that maybe doesn't want to use this service, simply hasn't heard about it yet, or accidentally publishes a whole archive of original articles they forgot to sign with the proof-of-authorship/timestamping API just before they get scraped by an ill-intentioned content farmer, that quickly uses the API to sign/stamp the articles to themselves before the content creator can. They don't even need to publish them right away, they can drip them out over years and have the proof-of-authorship signature to "prove" their authorship.

If we would actually trust this system, it means that the real original content creator is shit out of luck. There's no way they can prove their authorship in a way that distinguishes them from a scraper, that part remains the same with or without this system--but what would be new is that the scraping non-author now has some sort of extra claim of authenticity over the actual original content creator.

I see no way around this. Unless you devise a way so that all content written (anywhere, any time, on any medium) is immediately signed with a proof-of-authorship, such a system will be giving power to bad actors to claim authorship of any content that happens to slip through without being signed.

Google offers this service already:

http://www.labnol.org/internet/fat-pings-for-content-scraper...

https://en.wikipedia.org/wiki/PubSubHubbub

https://pubsubhubbub.appspot.com/

Most of the big hosted publishing platforms like WordPress and Blogger already use it, but it's pretty common for sites that built their own codebase not to.

(This was one of my interests at Google, and I had both a 20% project [unreleased] and a real project [Google Authorship] that were devoted to algorithmically detecting copied content and providing reliable attributions. Ultimate though, FatPing/PubSubHubbub [done by another team] was a much more robust way of doing this, with the downside that it's on the webmaster to implement it.)

Thanks for sharing this. I stand corrected (by a Googler, no less!). I wish Google pushed for greater adoption, then, as it does with Page Speed, SEO advice, etc.

That's not a timestamping service; that's just a way to quickly notify Google when you add a page to your blog.

What if your blog doesn't use Pubsubhubbub, but mine does, and I copy your blog post? Google will see my post first, but that doesn't prove I wrote it, so Google cannot use that to make ranking decisions.

I don't see how pubsubhubbub differs from the originally proposed timestamping service. For any timestamping service, it will still be the case that if you don't use it, and somebody else scrapes you and does use it, their timestamp will be better than yours.

There can be more than one timestamping service.

This is not an advantage.

Imagine a scenario where a rogue timestamping service performs just well enough to get a significant userbase. They then alter their service such that instead of simply returning the timestamp, they put the content up on the Internet, but assign themself an earlier signed timestamp before returning a timestamp to the original requestor. They would instantly replace all of their customers in Google results, because Google would think that they were the originators of the content. They'd then be free to do all sorts of mischief with that influx of traffic: show ads, install malware, phish addresses, etc.

The problem is how one prevents the bad-actors from taking advantage of such a service.

Define "bad actor".

That's not a "they aren't", but a request for "what do you consider to be acting badly?"

Conversely, is content scraping-and-publication possible by good actors?

How?

Timestamping sounds like a good idea. We can easily do it now in a decentralized way without depending on any 3rd party service e.g. https://github.com/chainpoint/chainpoint

Then if the original didn't use timestamping, and the clones did, they'd always win over the original.

Reply on news.ycombinator.com