Hacker News Reader

Comment by CaptSpify | Hacker News Reader

My favorite is "Oh, you can find the answer here: http://brokenlink.com"

At least half of every enterprise software I have used suffers from catastrophic link rot. And when you look at the URLs that do still map or redirect to something marginally on-topic, you just know it's a single API change away from redirecting to "we don't know what you're looking for, try searching here for hundreds of completely off topic (and occasionally broken) links".

No, I am not bitter. Learning where their FTP server is and how to navigate, well, that was gold.

This can often be solved by prepending

    https://web.archive.org/web/*/

to the now broken URL.

Hmm, there must be some extension that detects 404 errors and/or server unresponsive and prompts whether you want to check Google's cache or the Internet archive. That browsers haven't already integrated this (with a configurable list of archiving services, like search engines), is actually rather surprising to me.

There are extensions with that kind of functionality, but by now I have that sequence hardwired in my fingers.

There is: Resurrect Pages.

https://addons.mozilla.org/en-US/firefox/addon/resurrect-pag... https://github.com/arantius/resurrect-pages

There are many such extensions, but yes, we're working on getting directly integrated into Firefox, Chrome, Edge, and if anyone has a contact at Safari (or one of the other browsers), my email addr is in my profile. Thanks.

And then finding that the page is blocked by a robots.txt that is present now, but probably wasn't when the page originally existed.

Robots.txt is respected retroactively?

Yes, but it is only temporary. As long as you have a robots.txt file excluding some URLs, those URLs will: 1) not be crawled by the Internet Archive crawler, 2) not be shown in the Wayback Machine. Any already-crawled pages will, however, invisibly remain in the archive, and will reappear once they are not in the robots.txt anymore.

Reply on news.ycombinator.com