The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them).
The rendered HTML, on the other hand, is designed for humans, and so what you see is what you get.
But, his project really was very reasonable:
- it fetched ~2,400 pages
- he cached them after first fetch
- Wikipedia aggressively caches anonymous page views (eg the Queen Elizabeth page has a cache age of 82,000 seconds)
English Wikipedia does about 250,000,000 pageviews/day. This guy's use was 0.001% of traffic on that day.
I get the slippery slope arguments, but to me, it just doesn't apply. As someone who has donated $1,000 to Wikipedia in the past, I'm totally happy to have those funds spent supporting use cases like this, rather than demanding that people who want to benefit from Wikipedia be able to set up a MySQL server, spend hours doing the import, install and configure a PHP server, etc, etc.
For 1 person consuming from one of the most popular sites on the web, this really reads big.
The 2,400 pages, assuming a 50 KB average gzipped size, equate to 120 MB of transfer. I'm assuming CPU usage is negligible due to CDN caching, and so bandwidth is the main cost. 120 MB is orders of magnitude less transfer than the 18.5 GB dump.
Instead of the dumps, he could have used the API -- but would that have significantly changed the costs to the Wikimedia foundation? I think probably not. In my experience, the happy path (serving anonymous HTML) is going to be aggressively optimized for costs, eg caching, CDNs, negotiated bandwidth discounts.
If we accept that these kinds of projects are permissible (which no one seems to be debating, just the manner in which he did the project!), I think the way this guy went about doing it was not actually as bad as people are making it out to be.
In theory, you'd want to cache more popular pages and let the rarely visited ones go through the uncached flow.
Crawling isn't user-behavior, so the odds are that a large percentage of the crawled pages were not cached.
Checking a random sample of 50 pages from this guy's dataset, 70% of them were cached.
This amount of activity really isn't something to worry about, especially when taking the fast path of logged out user viewing a likely to be cached page.
How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?
Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT
It's a whole little clever language (https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa...) for making complex diagrams out of rather simple pictograms (https://commons.wikimedia.org/wiki/Template:Bsicon).
But every other infobox I've seen has key/value pairs where the key was always a string.
So what's the spec for an info box? Is it simply to have a starting `<table class="hello_i_am_infobox">` and an ending `</table>`?
If you want machine readable use wikidata (if you hate rdf you can still scrape the html preview of the data)
That plus a few other key things (categories, opening paragraph, redirects, pageview data) enable a lot of powerful analysis.
That actually might be kind of a neat thing to publish. Hmmmm.
Am I anywhere close to being correct?
That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
Not quite. Many Wikipedia infoboxes (and some other templates) use standardised class names from microformats such as hCard:
Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages.
It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).
Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every second).
They said please don't, not don't do it or they'll sue you.
But content license and site terms of use are different things.
From their terms of use you aren’t allowed to
> [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website;
Wikipedia is also well within their rights to implement scraping countermeasures.
Two things:
1) The English wikipedia *alone* gets 250 million page views per day ! So you would have to be doing an awful lot to cause "undue burden".
2) The Wikipedia robots.txt page openly implies that crawling (and therefore scraping) is acceptable *as long as* you do it in a rate-limited fashion, e.g.:
>Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please.
> There are a lot of pages on this site, and there are some misbehaved spiders out there that go _way_ too fast.
>Sorry, wget in its recursive mode is a frequent problem. Please read the man page and use it properly; there is a --wait option you can use to set the delay between hits, for instance.At a previous company we had the exact problem that we published all of our content as machine readable xml, but we had scrapers costing us money by insisting on using our search interface to access our content.
2. No one is going to jail for scraping a few thousand or even a few million pages, but just because low speed web crawlers are allowed to index the site, doesn't mean scraping for every possible use is permitted.
As a general rule, make your scraper non-paralell, and put a user-agent that has contact details in the event of an issue, and you're probably all good.
After all wikipedia is meant to be used. Don't be unduly disruptive, don't scrape 20 million pages, but scraping a couple thousand is totally acceptable.
Source: used to work for wikimedia, albeit not in the sre dept. My opinions are of course totally my own.
(Before anyone takes this out of context - no im not saying its ok to be intentionally disruptive, or do things without exercising any care at all. Also always set a unique descriptive user-agent with an email address if you're doing anything automated on wikipedia).
It’s incredibly easy for careless scrapers to disrupt a site and cost real money without having a clue what they’re doing.
I want people to think twice and consider what they are doing before they scrape a site.
I mean, sure, you can do a lot of things you shouldn't with freely available services. There's even an economics term that describes this: the Tragedy of the Commons.
Individual fish poachers' hauls are also, individually, de-minimis.