Hacker News Reader

Comment by pvg | Hacker News Reader

I didn't think anybody else would be that interested.

'Save the webpage as I see it in my browser' remains a surprisingly annoying and fiddly problem, especially programmatically, so the niche is probably a little roomier than you might initially suspect.

View on HN

> 'Save the webpage as I see it in my browser' remains a surprisingly annoying and fiddly problem

You may be interested in SingleFile[1]

[1] https://github.com/gildas-lormeau/SingleFile

I use it all the time to archive webpages, and I imagine it wouldn't be hard to throw together a script to use FireFox's headless mode in combination with SingleFile to selfhost a clone of the wayback machine.

Thanks, I've seen it, last I tried it it missed bg images. But my point is this is something browsers should support better and kind of sort of do now but even with that it's a hassle.

I tested this just now on the blog post that this HN page points to and SingleFile handled the background image fine.

> FireFox's

It's just "Firefox".

This is what I was going to say as well. Somebody on HN told me about SingleFile and I use it all the time now! Really amazing extension.

I've enjoyed using this

https://github.com/webrecorder

It has a standardized format and acts like a recorder for what you see.

Thanks all the JS - SPA develops that insisting on putting JS all over the place. Wouldn't it be better to have everything in one .html, using <script> <style> just inline. Then it is also just one file over the internet. There must be a bundler that does that no?

Seems JS developer just want their code to the obfuscated and unachievable as possible unless it is via their web server.

> using <script> <style> just inline

These SPA bundles are on the order of megabytes, not kilobytes. You want your users, for their own sake and yours, to be able to cache as much as possible instead of delivering a unique megablob payload for every page they hit.

Good point on the cache. However things such as putting background image in CSS, so user can right click to download the image is just stupid. Why is css all the sudden in control of the image display? It just makes archiving pages harder.

> 'Save the webpage as I see it in my browser' remains a surprisingly annoying and fiddly problem

Is it really? I remember hacking around with with JavaScript's XMLSerializer (I think) like 5 years ago and solved that for ~90% of the websites I tried to archive. It'd save the DOM as-is when executed.

Internet Archive/ArchiveTeam also worked on that particular problem for a very long time, and are mostly successful as far as I can tell.

90% feels like an overestimate to me but it's already quite poor, you wouldn't accept that for saving most other things. Another problem is highlighted in the piece - it's a hassle to ensure external tools handle session state and credentials. Dynamic content is poorly handled, the default behaviours are miserable (a browser will run random Javascript from the network but not Javascript you've saved, etc).

There's a lot of interest in 'digital preservation' and perhaps one sign of how it's very much early days of the field - it's tricky to 'just save' the results of one of the most basic current computer interactions - looking at a web page.

But if you serialize the DOM as-is, you literally get what you see on the page when you archive it. Nothing about it is dynamic, and there is no sessions nor credentials to handle. Granted, it's a static copy of a specific single page.

If you need more than that, then WARC is probably the best. For my measly needs of just preserving exactly what I see, serializing the DOM and saving the result seems to do just fine.

Yes you save something that's mildly better than print-page-to-PDF. But it still misses things and the interactive stuff is very much part of 'exactly what I see'. Like, a random article with an interactive graph, for instance - like this recent HN hit https://ciechanow.ski/airfoil/

It's not that there aren't workarounds, it's that they are clunky and 'you can't actually save the most common computery entity you deal with' is just a strange state of affairs we've somehow Stockholmed ourselves to.

> Internet Archive/ArchiveTeam also worked on that particular problem for a very long time, and are mostly successful as far as I can tell.

One category that the archivers do poorly with is news articles where a pop-up renders on page load which then requires client-side JS execution to dismiss the pop-up.

Sometimes it is easily circumvented by manual DOM manipulation, but that's hardly a bulletproof solution. And it feels automateable.

Print to PDF seems to be the only way to ensure you record what you saw.

Reply on news.ycombinator.com