Hacker News Reader

Comment by Cogito | Hacker News Reader

They mention at the end of the article that 'content drift' may be a bigger issue than link rot; when the content of the post is simply changed rather than missing, it is much harder to notice.

Is there a scalable way to monitor Wikipedia links to see if the content is changed after originally being posted?

They are already storing every link in the Internet Archive when it gets added, so there should be a reference point to compare against.

One easy option would be to make Internet Archive links available for every single link on Wikipedia, even if it hasn't rotted yet. So a 'live' link to the current content, and an archive link for what it was at the time of linking.

View on HN

Interesting aspect!

The biggest problem in this would probably be how to recognize if "content" changed. A site can change the full design, navigation, footer and header and everything and still have the exact same "content". For a human being this will be simple enough to understand, but a tool might have its problems with that.

Yes, this is a fundamental issue if you wanted to do this at scale.

There are a few solutions to this already, using solutions like outline.com to pull the content out of the cruft, but I don't know how many of these are general purpose and how many are purpose built for each site (and maintained for the current version of the site, perhaps?)

As seen in the article, most links are to a small number of sites, so perhaps hard coding the content extraction would be feasible, especially for an initial study.

It would be interesting I think to see just how many links have identical content, but you're right in that the number will be skewed greatly if there are any ads or similar included.

You might start with Wikipedia article's history pages.

Does the article's history page keep a record of what the content of the link was when it was updated?

I guess it doesn't, but assuming that IA records every link as it gets added you can use the history page to see which snapshot on the wayback machine corresponds to the link at the time it was added.

You could use that to compare to now, but unless you made sure to snapshot the content regularly (whenever the page is edited?) you wouldn't necessarily know when the content drift happened, or if multiple changes have happened to the content over time.

Using the history page would likely give you enough information to, for example, do a study on links to find out how many have different content now vs when the link was added.

Ah, no, that would be a separate problem.

Though IA's archival-on-creation mechanism should at least mean that the orighinal reference is preserved.

An explicit Wikipedia link to reference-at-time-of-archive, or better, a diff flag or listing, would be useful.

Reply on news.ycombinator.com