WikipediaQL: Query language for efficient data extraction from Wikipedia (early | Hacker News Reader

WikipediaQL: Query language for efficient data extraction from Wikipedia (early

github.com

37 pointstodsacerdoti12 comments

I really like this. I am into SPARQL so I prefer the WikiData SPARQL endpoint, but for most developers this looks like a good query tool.

Thanks!

I am mentioning Wikidata/SPARQL in README (TL;DR: less data still, harder to discover unless you are deeply into SemWeb tech), but I expect this project, will it live long (fingers crossed) will definitely has some relations with Wikidata, too. My previous attempt on something similar: https://github.com/molybdenum-99/reality — used both Wikipedia and Wikidata (and OpenStreetMap and...) but however cool were few demos I managed too achieve, it was a dead end for several reasons.

The crazy thing is that this is a full text search of sorts, not pre-structured Wikidata.

If you ever tried to scrape HTML, you know how untidy it usually ends up; this one looks clean.

This tool might be useful to Wikidata contributors. Getting data out of Wikipedia and into Wikidata takes a lot of effort.

Maybe use GraphQL instead of inventing a new query language?

I am (slightly) contemplating this (see also Reddit discussion: https://www.reddit.com/r/programming/comments/oe9oia/wikiped...). But currently I am mostly focused on deducing atomic selectors that would be useful and choosing their interaction. The particular syntax is kind of afterthought.

Though, I should say that I am still not sure GraphQL's mental model would work here (WikipediaQL for now structured around "which part of the document to choose and how to transform it", with "where to put it" being the final statement just of some branches; with GraphQL, as far as I can understand, each node is "where to put it"-first, and it _may_ lead to clumsy and repetitive queries).

Is Wikipedia available as an SQL database dump?

That would be really useful.

Yeah of course, downloadable als SQL or XML: https://en.wikipedia.org/wiki/Wikipedia:Database_download#En...

Yes and no. You can get SQL dumps (although XML is more common), but it doesn't contain any useful structure. The article content is just wikitext, and only metadata such as revisions are structured.

You can get a lot of data from Wikidata. They have a SPARQL interface at https://query.wikidata.org/#%23Humans%20born%20in%20New%20Yo... and you can also get all the data in a variety of rdf formats.

You can get all the edges (links between pages) and nodes (pages and text therein), so not sure what you mean about structure. I loaded all of Wikipedia into Neo4J like 6 months ago... Wasn't very hard.

Can you extract from wiktionary with this?

The answer is complicated (I am mentioning it in the midst of plans and possibilities in README: "(maybe?) other MediaWiki-based sites (like Wikvoyage, Wiktionary, etc.)").

In more details:

1. With the current library, you can't (connection to en.wikipedia.org is hardcoded :shrug:)

2. Adding "what MediaWiki instance to connect" is relatively easy (they all have the same API), including Wiktionary

3. For Wiktionary, current (v. 0.0.2) selectors will be useful, but future ones (infobox, wikitable and other high-level Wikipedia-specific stuff) will be not

4. So I might think of some way of specifying "set of selectors relevant for this instance" or something like that.

5. So, in some distance future, Wiktionary definitely would be fully supported; how distant—probably depends on how fast the project will be moving forward, and how it will be (hopefully) used by others

6. But I might add a support for "use this instance of MediaWiki" in the near future, it is easy!

Reply on news.ycombinator.com