Hacker News Reader

Comment by falcolas | Hacker News Reader

My favorite use of wget: mirroring web documentation to my local machine.

    wget -r -l5 -k -np -p https://docs.python.org/2/

Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation.

I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects by default, `curl -O` will not. Yes, I could remember yet another argument to curl... but why?

That said, I love curl (combined with `jq`) for playing with rest interfaces.

View on HN

Thanks for the jq tip. I hadn't seen that before. Link if anyone's interested:

https://stedolan.github.io/jq/

Underscore CLI looks interesting as well, though I haven't personally had much of a chance to play with it. It does require NodeJS, which might be a deal-breaker for some, but if it's already in your toolchain then it might come in handy.

https://github.com/ddopson/underscore-cli

I've written a similar tool underscore-cli and jq called jowl. It's designed to be easier to learn (for JavaScript developers) than underscore-cli or jq. The README includes a comparison.

It's still early in its development, and would benefit from a tutorial and a few more features, but it's getting there.

https://www.npmjs.com/package/jowl

N.B., this will make curl redirect by default:

  echo "-L" >> ~/.curlrc

This is a great use, and I've used it for a lot of other sites and documentation. However I just want to point out that Python's documentation happens to come in an easily downloadable archive of HTML files:

https://docs.python.org/2.7/download.html

(you can also find archives for Python 3 and SciPy and NumPy)

wget is ludicrously good for mirroring. I use it to mirror the entirety of EDGAR.

For my money another tool that belongs in the toolbox is perl's WWW::Mechanize and its component WWW::Mechanize::Shell.

How does wget compare with httrack for mirroring?

It's somewhat hard to compare CLI vs GUI programs.. but in my experience, wget --mirror is simpler than HTTrack for simple tasks, which takes a little bit of mucking around to start downloading. HTTrack has a smaller learning curve for more complex situations though.

For docs specifically, you could also try Zeal, which allows you to browse documentation for quite a few programming languages offline, including Mozilla's JS docs and an HTML spec, as you mentioned in some other comment in this thread.

https://zealdocs.org/

Wow. This is so incredibly useful to me.

The general workflow I have followed was, "Download PDF if present. If documentation can be shown in single HTML page, download it, change all the anchors to point to my own filesystem". This changes everything in such a simple, useful, way.

I bet that would be extra useful for creating Dash.app docsets...

if you look at some of the default docsets you can tell that this was most likely done. All of the directories under the file root in the docset are named after domain names (one of the things whet does when crawling a site and traversing multiple domains).

Or just curl -OL

Is this the part where I tell you about pydoc and blow your mind?

I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.

Sure. Does it work with golang docs, the w3 HTML5 spec, Mozilla's JS reference, etc?

;)

Python's built in documentation tools are good, but the command I referenced also gives you the full language spec locally, as well as the FAQs and packaging docs.

Reply on news.ycombinator.com