Hacker News Reader

Comment by tomcam | Hacker News Reader

It’s a truly distinct and amazingly designed website that I’m often not smart enough to understand. It’s a unique property on the Net, and that’s saying a lot these days.

One thing I would like to understand more is the archive system (used to reduce the problem of dead links). Is there some kind of automated process to create local copies of the linked webpages, or is it done by hand using some thing like wget or curl? How far do they go to archive embedded assets such as images or PDF files? Are there any copyright considerations for archived articles (what if you archive someone’s article that they would prefer to keep off-line)?

View on HN

There is some info about archiving here: https://www.gwern.net/Archiving-URLs

Lots of jobs and scripts, plus usage of archive.org as well. It's an interesting read.

Just what the doctor ordered. Thank you.

That page is a bit outdated because I am still finetuning the on-site archive system before I do a writeup.

I still use archiver-bot etc, they're just not how I do the on-site archives. See https://github.com/gwern/gwern.net/blob/master/build/LinkArc... https://github.com/gwern/gwern.net/blob/master/build/linkArc... for that.

The quick summary is that PDFs are automatically downloaded, hosted locally, and links rewritten to the local PDF; other URLs, after a delay, call the CLI version of https://github.com/gildas-lormeau/SingleFile to run headless Chrome to dump a snapshot, which are manually reviewed by myself & improved as necessary, and then links get rewritten to the snapshot HTML. They get some no-crawl HTTP headers and robots.txt exclusions to try to reduce copyright trouble.

THANK YOU for scratching that itch.

There are plenty of rabbit holes to get into when it comes to web archiving, but for blog authors that just want to preserve a page referenced by their text, using a tool like SingleFile is a great first step.

https://github.com/gildas-lormeau/SingleFile

Def a good start, thanks.

Web archiving goes pretty deep, actually. I have no idea what this site uses but the open source tools maintained by the Webrecorder project are rad.

Thanks for that reference. I didn’t know about it until you posted. Note to downvoters: that was a sincere comment.

... downvotes? Weird.

There's other cool web archiving stuff out there too that's just the one I know about.

It went into the negative very quickly. For some reason “thanks for the reference“ must’ve come off as snarky, probably because I was too stupid to find it myself but others weren’t!

Psh. People. So judgy.

Reply on news.ycombinator.com