Hacker News Reader

Hacker News

Comment by toomuchtodo | Hacker News Reader

Parent Full thread

toomuchtodo

You can always package your site up in a format Archive.org can read:

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...

http://justsolve.archiveteam.org/wiki/WARC

http://warcreate.com/

View on HN

Thanks for the info, was unaware of ARC/WARC formats. That said, I still think many people are looking for something simpler/easier, and a daily screenshot is good enough. Particularly, it will guarantee preserved formatting as browsers continue to evolve.

I agree that screenshots are easier; WARC files are future-proof.

You can probably make your service do both screenshots and WARC, instead of loading a site directly, load it through WARC Proxy (https://github.com/odie5533/WarcProxy), that will write out a WARC file and you can still store your screenshot.

Once you have the WARCs you can upload them to Archive.org and they can be added to the wayback, or you can set up your own service for browsing them, built off something like warc-proxy https://github.com/alard/warc-proxy (Yeah, same name different purpose...)

There is also a MITM version of WARCProxy that will let you store HTTPS sites: https://github.com/odie5533/WarcMITMProxy

As of version 1.14, wget natively supports warc (including built-in gzip and cdx index file generation).

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

This makes creating a browse-able mirror of a site in warc format fairly straightforward, as wget will automatically make links relative, as well as fetch requisite files (css, js, images) for each page.

Yeah, but as far as I can guess, derwiki's service doesn't use wget, so running a proxy to store the WARCs is the next-simplest thing.

If his service runs on any sort of Linux distro, its stupid simple to call wget with a system call. Wget comes standard with all of the most popular distros.

Reply on news.ycombinator.com