Comment by benten10 | Hacker News Reader

Comment by benten10 | Hacker News Reader

Parent Full thread

Here's a startup idea that I'd been holding close to my heart, but now that my teammate's working for a BigCorp, earning beyond her dreams, it's time for it to go public. Steal this idea!

Link rot is only going to get bad from now. It's real, and it's awful.

It's not even random websites. NYT will post a link to a popular Youtube video, two days later the video gets pulled, and one week into the article, the links are already stale.

As Wiki gets more 'reputable', newspapers will posts links to it. Wiki being what it is, the reference to the page will go stale.

So here's an idea for a service: hash all the outgoing/inside pages on a website. If they change, either 1) give options for users to review, 2) update the link 3) delete the reference. If the original website 404's, change the link to the archive page. If a video link is outdated, provide tools to search for similar videos to link, from inside the dashboard. For Wikipedia, automatically link to a certain version of history of a page, not the general page. This is different from the idea posted below because it integrates with existing publishing systems, so it'd be more b2b, and one would start cashflow right away.

For links to social media, use a 'photo snippet' tool that looks to check if the link is valid, and goes to the image version of the link if validity is dead.

I'm certain people would pay GOOD MONEY for this service. I know I would if I were running a publication.

Give me some spare change if it succeeds. : )

You can't simply hash the whole page, because for most sites the hash will change constantly due to any varying content (e.g. rotating ads, new comments, random messages, A/B testing...). I wrote up a proposal for using a CSS rule to select only the "relevant" portion of a page to hash.[1]

[1] https://bentrask.com/?q=hash://sha256/2c9e53858b5312564a2b8f...

The ultimate version being complete capture of the Internet at large with hour to hour snapshots and history perhaps using deltas between versions.

Git Internet!

I've personally started saving sites I want to keep using the print-to-PDF feature. Bookmarks aren't enough when you really care to save the data.

I use the Firefox version of Zotero[1] to archive links. It works quite well, and is more searchable than PDFs.

[1]: https://www.zotero.org/

You can save a lot of space by using a "reader-view" version of the webpage.

Some friends of mine built https://preserve.io to automate that. It's basically bookmarks-plus-print-to-pdf-as-a-service.

You've just described IPFS: https://ipfs.io/

Thanks for the link! That's a pretty interesting project I was previously unaware of.

You're welcome!

I hate to say it, but you've just described what a good SEO does. The good SEOs (and SEO firms) will analyze and take care of your internal links, your internal link structure, and make sure the site isn't linking out to outdated/old links.

If you want to do this yourself, there's several crawlers out there that you can find the data yourself.

Many publications have been reluctant to link out to sites over the years because of this very thing--link rot.

Good SEO still doesn't fix the root of the problem--lost content. If you write an article about a video, and that video gets pulled, there goes your article.

> As Wiki gets more 'reputable', newspapers will posts links to it. Wiki being what it is, the reference to the page will go stale.

Unless the page was non-notable and got deleted, there'll be a redirect.

I found myself so enraged at what Wikipedia privileged editors considered non-notable that I haven't had the energy to contribute in the past few years. I'm not talking about my 2nd cousin's garage band: see the talk page on reStructuredText (which managed to survive).

I don't understand why you were enraged. The editor who spoke with you made it very clear what was needed to prove notability (reliable secondary sources) and politely explained things to you while linking to official policy. I'd hardly call them privileged. It'd be another story if they said something like "I don't want this article on my encyclopedia"; he even recommended using the IBM DeveloperWorks link. The tag that was on the article is merely a signal to editors that the article needs better sourcing. To be deleted, it'd need to be nominated, where it would need to be proven that the topic is non-notable, so the system actually works in favor of keeping articles that are actually notable. It might seem like the rules are arbitrary or wielded by wikilawyers but if you read them they're not that bad. I don't agree with all of them, but "when in Rome..."

I was the one that found the IBM link. The notability skeptic liked it, but not enough to think that was sufficient for notability. After I gave up, someone with more influence came along and bigfooted the issue. I have no visibility into why one person's opinion is more important than another's - I assume it's because they've done something like devoted 10,000 hours to editing - but I've got no time to for environments that value fanatacism over reasoned argument.

And the Tool Support section - the most valuable part of the page, and which I vehemently contend constitutes essential secondary source material - remains deleted. It took me 15 minutes of digging through the page history to find it, which I'd never do if I didn't already know it was there.

The great challenge is proving something is notable. If contested it can be an uphill struggle, because journalists don't talk about obscure things much.

That's the whole point of Wikipedia. Otherwise it would be like PR Newswire.

Reply on news.ycombinator.com