Hacker News Reader

Comment by jl6 | Hacker News Reader

Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.

View on HN

I couldn't agree more. I wish more sites would load without needing multiple seconds of JS execution and AJAX. One of my TODOs is to get full-page screenshots working as well.

in some cases it leads to worse horrible experiences like my Linkedin feed. I have to wait a few seconss everytime before content loads.

> One of my TODOs is to get full-page screenshots working as well.

https://medium.com/@dschnr/using-headless-chrome-as-an-autom...

You might find this useful if you use Firefox:

- shift+f2

- screenshot urcodiaz.png --fullpage

Thank you for this tip! I have always installed Abduction plugin if I needed this, but the less plugins the better.

PDFs are really not suitable for archiving websites since they're designed around pages and the web does not have pages.

A better option is to render a page with JS turned on and save the resulting HTML.

PDFs have the advantage of being a fixed format that should display the same everywhere. It's probably less fragile than modern CSS or HTML.

However, as you point out, PDFs are designed around the printed page, not the flowing arbitrary-page-size documents of the web.

I was doing this before with the chrome --dump-dom flag but the output I was getting was garbage, and no more useful than the simpler wget download. PDF turned out to produce really nice, readable archives about 75% of the time, so I kept it in. Text-based sites tend to do a good job of having PDF-friendly styling.

I googled it, and --dump-dom simply dumps the output of document.body.innerHTML. I have not used headless chrome at all, but I imagine it would not be that hard to get it to dump document.documentElement.outerHTML instead.

(execute JavaScript on the page is most likely possible?)

I've been trying to do something like this in this project: https://github.com/ianb/pagearchive

It came out of a screenshot/archiver I've been working at at Mozilla, but I've split it up as the screenshotting is shipping and DOM archiving is still way outside Mozilla's comfort zone.

Wouldn't a copy of the DOM be even better than a screenshot?

I.e. DOM copy > screenshot > wget?

I mentioned it below, but I tried getting DOM snapshots using chrome --dump-dom, but the output usually didn't render well without a <head> section. (chrome only outputs the <body>) I could attach the headers from the wget file... but then it starts getting messy and complicated.

Agreed. I research new media and archive.org is invaluable to me. I worry that current web sites won't be able to be preserved. (much like many of the flash sites and real audio of the past are largely gone.)

Speaking of audio, one of my highest priority TODO's for this project is to use youtube-dl for nicely archiving youtube & soundcloud links.

But what do you do when a website has a broken media query, essentially destroying the print layout? Then a PDF is useless.

Well, I took a screenshot, better than nothing.

Reply on news.ycombinator.com