I'm curious what the strategy is to get companies like Google to pay for it. Is it "all carrots" or are there any "sticks"? Like maybe rate limiting certain consumers?
So the option you have is using stale data dumps. They're reasonably workable, I even host low-markup versions of every wikipedia article myself in my home lab and it's like some 100 Gb worth of compressed data, images excluded. Uncompressed I don't know where it ends up, but below 1 Tb at any rate. Manageable but very unwieldy. File systems like ext4 struggle with that number of files in a directory so you need to be a bit crafty, but it's not hard to get around. If all you want is an info box, you could probably squeeze it down to a couple of hundred gigabytes uncompressed. Definitely not something you need a data center for.
The catch, as mentioned, is is that the data is a stale snapshot. The only way you are getting current data is on the main site, which may be a problem if you have ambitions of staying up to date. I personally don't mind. I figure they used to sell printed encyclopedias and that worked pretty well so having a 1 year old copy of Wikipedia isn't that bad.
It's also a sort of weird domain. I can see how it might not be easy to host in the cloud in a way that's both affordable and performant. Since so many insist on cloud-based apps nowadays, there's probably a market for Wikipedia APIs with a good SLA.
It does cache pages very aggressively. And it even has solutions for hard problems in computer science, cache invalidation.
Pages are cached indefinitely until something pokes it.
Proof? Add ?action=purge to a wiki page.
Like for example this: https://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks#Re...
and this: https://meta.wikimedia.org/wiki/Live_mirrors
Something resulting in 50k page fetches of cached data in a day is nothing. A raspberry pi could handle 50k static documents in an hour.
The 50k daily hit on https://meta.wikimedia.org/wiki/Live_mirrors is merely to illustrate a web proxy can dramatically reduce the number of hits to the original site. They could have pick 42 or 7 billions, it is just an example.
The text dates from 2009, and in April 2009 this is roughly what we had in term of servers https://commons.wikimedia.org/wiki/File:Wikimedia-servers-20... Arguably a little more than a Raspberry pi
We already had a few million pages rather than 50k of them, and they were not static since pages get edited and made live as soon as one save their edit. Obviously, we already had million daily users and definitely requires a ton of caching all across the stack. Also good luck serving flat files that keeps being written too, that does not really work with hard drive and static files storage.
In September 2021 we have served 21 billion pages and 73 billions media files https://stats.wikimedia.org/#/all-projects . Almost all of them served primarily served from caches.
That being said, the wiki pages you have mentioned are a decade old. We definitely had trouble with people having the smart idea of scrapping the whole website, often using either uncacheable content or hitting barely served articles or history that were not cached and thus caused a full rendering of the page. Even with the servers we had, a single user could cause major havoc on the infrastructure and keep a good part of them busy just to serve that single person. Hence the recommendation to use the database dump which are still used: https://dumps.wikimedia.org/
cheers
Don’t worry, they are hard at work at making workarounds for all their idiotic design decisions that seem to make absolutely no sense.. and those workarounds will be provided quite cheaply!!
Reminds me of the saying: “ It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It”
We can be quite sure that Wikipedias engineering will never improve if their income becomes dependent on providing workarounds for deficiencies in their solutions.
I never found it to be a big deal. Once a day updates were perfectly fine for reference usage.
Google USED TO also have lots of prominent links that would drive traffic (and hence, donations) to Wikipedia. Over time, it has removed a lot of those links and replaced them with ones that lead to more Google hosted content.
There's a distinct drop in Google-referred traffic to Wikipedia over time that's in stark contrast to the increased number of eyeballs on Wikipedia sourced content on Google's pages.
It's not clear to me whether Google's sponsorship reflects the value they've gained. And, yes, I get that they are under no legal obligation to do anything at all, since the content license allows for all of it. But as with rich snippets, you can't just "take", or eventually, there won't be anything to "take".