Show HN: Terminal Based Wikipedia | Hacker News Reader

Show HN: Terminal Based Wikipedia

github.com

138 pointsdarksoul101179 comments

This is great! For my purposes using a downloadable Kiwix snapshot would usually be better than scraping (not "scrapping", that's a misspelling) the live site. The usual downside of Kiwix is that the snapshots don't include many images, but that's less of a problem in the terminal!

incremental downloads for Kiwix is also very much missing to me.

I once did something similar at a job that bored me. I wanted to at least read wikipedia while pretending to work.

My go to at a job that involved maps was to stare aimlessly at the world and zoom in on random places that either looked interesting or at random. I learned a lot about lots and lots of tiny remote islands once I found their wikipedia pages.

May I ask how did you go about writing it?

i wrote it with Ruby, one parameter for the key word, grab wikipedia page, then parse some html and print some text out. I think I also used the edit page because it was easier to parse, no html just Wikipedia syntax.

Related: surfraw https://gitlab.com/surfraw/Surfraw

I'm a Python3 noob so I am not sure what I might be doing wrong, but after following the install instructions and trying to run it, I am getting this:

  Traceback (most recent call last):
    File "/usr/bin/fetch", line 33, in <module>
      sys.exit(load_entry_point('yashsinghcodes==1', 'console_scripts', 'fetch')())
    File "/usr/bin/fetch", line 22, in importlib_load_entry_point
      for entry_point in distribution(dist_name).entry_points
    File "/usr/local/lib/python3.7/dist-packages/importlib_metadata/__init__.py", line 947, in distribution
      return Distribution.from_name(distribution_name)
    File "/usr/local/lib/python3.7/dist-packages/importlib_metadata/__init__.py", line 538, in from_name
      raise PackageNotFoundError(name)
  importlib_metadata.PackageNotFoundError: No package metadata was found for yashsinghcodes

even though the package says 3.6 and up for Python, its using importlib.metadata which was introduced in 3.8. https://docs.python.org/3/library/importlib.metadata.html

You're using 3.7 here.

Thanks!

I'm running Debian so... of course my version is old.

Looks like a merge conflict wasn't properly fixed: https://github.com/yashsinghcodes/fetch/blob/main/test/test....

I'm not seeing it. Looks like a regular asset on a multiline string to me. Can you clarify?

I think it was fixed with this commit - https://github.com/yashsinghcodes/fetch/commit/0e0c59b4d24da...

Good old gopher & lynx days ...

As a side note, Wikipedia doesn't look that bad with styles turned off. I kind of like it because it makes the text body distraction-free; there's no content on either side, and it's more academic looking for lack of better words.

If you have Firefox, go to View > Page Style, and click on "No Style".

The only downsides are that the search box is less convenient, and there's no preview-on-hover.

If text-based access to popular websites is your cup of tea, there are two very good gopher interfaces I use regularly: gopher://gopherpedia.com for Wikipedia and gopher://gopherddit.com for Reddit.

The latter is not very helpful on image-based threads but is excellent for text-rich threads (like askreddit). The gopher interface to wikipedia is surprisingly full-featured; I love it.

Trying gopher://gopherpedia.com with a page linked on HN: https://en.wikipedia.org/wiki/Belling_the_cat

Any ideas why it's missing the first paragraph? I didn't spot a page revision without it.

Looks like the first paragraph was a recent edit. I'm not sure how gopherpedia works but perhaps it only syncs periodically.

That was my thinking too but I can't find a revision without it on the first page of results going back to 2019. Why does it look like a recent edit to you?

I don't have time to look too closely into it, but my guess is that it's a bug somewhere in the parser: https://github.com/muffinista/gopherpedia.com/blob/main/pars...

The markup behind that article isn't terribly complicated, but the parser isn't perfect either. For example: https://gopherpedia.com/gopherpedia.com/Gustave_Dore has some glitches in the lead sentence.

My suspicion was that it's getting confused by the line above the intro, but I pasted the article's source code into Wikipedia:Sandbox and gopherpedia read the intro just fine. Strange!

Aha! Here we go. Compare:

- https://gopherpedia.com/gopherpedia.com/Belling_the_Cat

- https://gopherpedia.com/gopherpedia.com/Belling%20the%20Cat

Without underscores in the URL, the first paragraph disappears. Bizarre.

Is there a gopher client for macOS? I haven't been able to find one. Command-line is preferred, but beggars can't be choosers.

`lynx` still works for me.

Oh, wow. I didn't know that Lynx does gopher:// URLs. Thanks!

I see MacPorts includes the following clients (I took the liberty of removing servers and proxies from this list):

    $ port search gopher
    bombadillo @2.3.3_2 (net)
        Bombabillo is a non-web client for the terminal, supporting Gopher, Gemini and much more.

    kristall-devel @20211120 (net)
        Small-Internet Browser for Gemini, Gopher, Finger & HTTP

    phetch @1.1.0 (net)
        A terminal client designed to help you quickly navigate the gophersphere.

    vf-1 @0.0.11_1 (net, python)
        command-line gopher client

What would the respektive emacs mode be ?

eww (built-in browser) works well.

There's also a package "emacs-w3m", an interface for the w3m web browser: https://github.com/emacs-w3m/emacs-w3m

Nice. Would be cool to have shortcuts numbers next to the links to navigate to linked articles.

Thanks for the feedback. I will add that in my next commit.

By the way, you are including a lot of useless files and binary files in your git repository. (.egg folder, pycache, pyc files). They make your repo very heavy. Take a look at this example .gitignore : https://github.com/github/gitignore/blob/main/Python.gitigno...

I still remember WikipediaFS from almost two decades ago when FUSE-based filesystems were popular (and that project allowed editing Wikipedia too).

My version of this is a bash script that opens the article in lynx:

https://github.com/boris-marinov/bin/blob/main/wiki

Seems like __version__.py is missing after a recent commit.

Seems like yes:

$ python3 setup.py install Traceback (most recent call last): File "/home/x/git/fetch/fetch/setup.py", line 12, in <module> exec(open("fetch/__version__.py").read()) FileNotFoundError: [Errno 2] No such file or directory: 'fetch/__version__.py'

I'm really sorry i was actually removing the __version__.py but forgot to remove the code from setup.py. You can try again.

It is fixed, thanks.

Nice work. Support of different languages would be neat!

dig <wiki url>

What `dig` is that? Surely not https://en.wikipedia.org/wiki/Dig_(command)

it's maybe one of the wikipedia-over-dns service. i had this but it does not work by now: dig +short txt Article_Title.wp.dg.cx

Wikipedia has an API. This is just fetching the HTML and parsing it with beautifulsoup, which seems like a terrible idea when an API is available.

They could have used https://github.com/wikimedia/pywikibot (python API interface) or https://github.com/goldsmith/Wikipedia

Website APIs tend to sprout "API key" requirements with little or no notice, which seem like a terrible idea. This kind of breakage is far less likely to happen to the HTML endpoint, because it gets about a zillion times more usage.

It is actually sensible to stick to the interface that the majority of users are using, because that interface is the least likely to break. Spontaneous API key hoop-jumping is a form of breakage.

Are you saying that it's more likely API key requirements appear at some point than the interface changing?

I'm not that informed, but intuitively, I'd doubt that.

As someone who's reasonably informed on this stuff, I would be absolutely shocked if Wikipedia ever introduced that requirement.

This is an example of good (well, ok) general advice, which absolutely does not apply to this specific instance.

As general advice it's only ok, because a) the details of served HTML can change a great deal and b) some APIs might have this problem, some are unlikely to. Wikipedia is an extreme example of the latter.

Was thinking the same thing...

It's probably easier to parse the HTML than the Wiki markup.

Useful, although the name is very unfortunate. It collides with a common installed by default tool in BSDs, and doesn't really suggest Wikipedia either.

I'd suggest wp as prefix in the command name. Eg: wpget, wpfetch would work.

Then you're clashing with WordPress. Something like wikifetch is even better I think.

"wp" and "wiki" are both unclear... Which "wiki"?

Just call this "wikipedia".

But what if support for other wikis gets added?

But what if it starts supporting other webpages too, not just wikis?

Let's just name stuff for what their stated purpose is, instead of trying to invent "maybes" and name it after that.

Then call it weboob.

I am already out --> []

What about `wget` then? (wiki* + get)

Yep, definitely no clash there. ;-)

wordpress isn't part of any base system, at least.

It's just some PHP (eww) blog software.

What anned20 means is that quite a lot of managed hosting has the (incredible) WP-CLI installed:

https://wp-cli.org/

The (completely invaluable) shell command is "wp".

eww? like the emacs builtin browser? ;)

MediaWiki is also written in PHP?

mwfetch (MediaWiki fetch)

Morrowind fetch à la OpenMW? :)

Kidding - although an API for Morrowind would be neat...

That's a decent name.

Thank you so much for the feedback. Actually i started it has a personal project for my linux machine but my friends suggested me to push it to the github and i did not know about BSD already fetch(same name). I will definitely change the name after this submission end(as it will break the link). As for now you can directly run it from fetch folder with python3. All name suggestions are welcome.

wik

Man, are you going to be annoyed to hear that I'm working on a personal knowledge base that I've decided to simply call "notes".

I guess it will not go well with FreeBSD ;)

https://www.freebsd.org/cgi/man.cgi?fetch(1)

Yeah, it's weird this command doesn't have anything that references Wikipedia in the name too. eg wikifetch or even just wpfetch. It's such a generic name for a specialized too.

Yes, that was my first thought: I'd better not install that, surely it'll conflict with something...

wp sounds like WordPress though.

Maybe mwfetch (MediaWiki fetch)?

Also, the name "fetch" has been taken... famously... for quite some time... https://en.wikipedia.org/wiki/Fetch_(FTP_client)

Should have named it "wordle," instead. jk. But there are not a lot of w-names. For example, "wi" is not taken, seems the most obvious name for this, and is very attractive for the command line; "wik" is almost as good. I wish I was a fly on the wall when whomever came up with this name said, "yeah, we'll call it Coca-Cola! That's original!"

Thanks for the suggestion. At the time of building the project i did not think much about the name many people has suggested some names and i will go with one of them after this submission stop ranking on HN. As changing it will break the link.

I'm using w3m as my terminal browser. What do I need a specialized wikipedia browser for?

If you're a w3m user, you must surely know that w3m usage is not suuuper common right? Maybe, just maybe, this project is not for you. Not everything is for everyone.

Sure. You can also use lynx. Both browsers already solve the problem of browsing text based web sites on the command line. I am just wondering why you'd want another one, that isn't even a regular browser.

I'm sure lynx + w3m together hold less than 0.01% of the global browser usage.

Why would you even want lynx or w3m when curl/netcat already exists and allows you to view text on the commandline from http/s servers? Because it's more comfortable for you. Just like this tool is more comfortable for you if you just want to visit Wikipedia articles in the commandline.

Some tools are specialized, I'm not sure why you think that's such a bad thing. Otherwise we'd all use netcat to view websites, then you can read most protocols too, not just http/s.

It seems to me that they're explicitly asking how this tool is more comfortable to use than w3m. Not sure how browser market share has anything to do with that.

This tool being specialized to do this one job doesn't mean it's automatically better at it than one with a wider range of functions. For example, in a Wikipedia-only browser you can't open references (without using another application) which arguably results in worse UX.

Use Lynx instead?

Reply on news.ycombinator.com