Wikimedia Enterprise – APIs for LLMs, AI Training, and More | Hacker News Reader

Wikimedia Enterprise – APIs for LLMs, AI Training, and More

enterprise.wikimedia.com

222 pointsks2048161 comments

While i am sympathetic to wmf finding alternative funding streams, I do get nervous about these sort of things due to the inherent conflict of interest and incentives to canabalize the free offerings. I'm not saying that is happening now, but will it happen eventually?

Additionally, originally it was promised this would all be open source, and officially they are sticking with that, but it seems like they are going with the model of throwing code over the fence like once a year, which does not really meet my expectations.

I’d rather Wikipedia offer it versus a for profit enterprise using their dataset. Cut out the middleman and vertically integrate.

There's a whole statement-of-principles thing that at least implies that the intention isn't to cannibalize the existing offerings: https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/Princip...

Though I imagine that only works so far as you feel you can trust the Foundation to stick to those principles, so that's complicated. :D

There's also a bunch of FAQs here that sort of get at how the funding streams are supposed to integrate into the existing structures and how it's supposed to avoid pushing out the free services: https://meta.wikimedia.org/wiki/Wikimedia_Enterprise/FAQ

(As I said elsewhere, I work for the WMF but I don't work on anything related to this so I'm just commenting as someone who has more experience searching through our public info than most HN commenters would...)

> Though I imagine that only works so far as you feel you can trust the Foundation to stick to those principles, so that's complicated. :D

At the end of the day, WMF is made up of people, and people follow incentives. I'm not saying they are bad people, but they aren't saints either. They are just people like anyone else.

It might not happen today, but 5 or 10 years from now, i'm not so sure. Eventually there will be some situation where people involved will have to chose between something for the public good vs something that sells enterprise APIs better. If WMF becomes dependent on the enterprise money, it will be hard to chose the public good. When that day comes, the enshitification begins.

After all, google once claimed not to be evil. The motto didn't last.

> While i am sympathetic to wmf finding alternative funding streams

Why are you sympathetic to it? Their fund, at this point, can make enough off interest on a basic CD to not just pay for every possible cost they could have until the end of time, but make the maintainer(s) obscenely wealthy without breaking a sweat.

https://upload.wikimedia.org/wikipedia/foundation/3/3e/Wikim...

$250m - they’re doing this out of greed, not need.

Over the long term of many years you're /lucky/ if a stable very-low-risk investment can net ~3% when accounting for inflation. Thus $250M could maybe net you roughly $7.5M/year. Exactly how many network links, servers, and engineering staff do you think that buys? It's way under what it operates on today, which is way under what it ideally should be for site like Wikipedia. And that's /just/ the operational engineering of the sites on a technical level.

You also need HR, you need Finance, you need a lot of Lawyers, you need software developers, you need a travel department, a fundraising team, PR people, community relations people, grant-making for the extended open ecosystem around the Wikimedia movement, conference planning, and the list goes on.

You're off by enough to seem troll-ish at best.

What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries.

Wikimedia's expenses are almost ENTIRELY going to staff. Their balance sheet for 2023 included $101m in expenses for salaries and benefits out of a total expense of $160m. Their hosting was $3m. So yes, I'm confident their network links and servers cost almost nothing, and they don't need anywhere near $101m in compensation to keep the lights on when the VAST majority of their content is contributed for free.

https://wikimediafoundation.org/wp-content/uploads/2023/11/W...

You may have missed them saying "when accounting for inflation". In the US at the moment that's around 3%. Thus your local credit union's savings account, a nice and stable investment, is effectively giving you around 3% appreciation in real-money each year right now. (I have no idea whether their broader point about the rate over-time is correct, admittedly.)

> Their hosting was $3m

Its kind of unclear what this includes. Computer equipment is a separate line, and wikimedia owns its own servers, so presumably that is separate from server costs. You don't have to buy new servers every year so some servers might simply have been purchased in other years, although maybe that gets ammortized, i dont know.

Additionally when you host your own servers you need staff to operate them. When using something like AWS, this would be part of your AWS fees, but if you operate your own servers then you have to pay that part separately. Its probably cheaper overall in the end when you are wikipedia scale, but the costs break down differently.

> What are you talking about? The AVERAGE CD right now is 5%. My local CU is almost 6%. US bonds are currently ~4.5% - if you consider those unstable, I guess the US economy isn't stable - and if the US economy crashes, wikipedia will be the least of their or our worries.

If you want to live off the interest you have to worry about inflation which essentially devalues your pot by x% per year, so if you really need y% for running costs you really need about x*y% to do it long term.

Hosting means nothing without the staff. Hardware, networking, datacenters, etc are the cheap part because the staff are good at their jobs.

You and the other set of trolls that think that Wikimedia can run itself need to appreciate that just because you work for a non-profit doesn't mean you should work for slave wages, or that you should be forced to work with the bare minimum amount of staff to keep things running without being able to make improvements to the infrastructure, reader experience, editor experience, or data consumer experience.

In comparison to similar services, Wikimedia has a relatively small overall budget that's well spent.

An engineer costs $500k a year. Salary, benefits, office space, equipment, hr, legal, and other overhead. The engineer will only see a fraction of that, of course.

If you told me it took a hundred engineers to run Wikipedia I'd say, that's not totally unreasonable. Features, design, api, scaling, moderation, there's a ton for engineers to be doing.

Your overall point still stands, but FWIW Wikimedia pays less than its peers. To compare two active listings

Senior Security Engineer at Mozilla (https://boards.greenhouse.io/mozilla/jobs/5803609): $124,000 to $199,000 plus bonus

Senior Security Engineer at Wikimedia (https://boards.greenhouse.io/wikimedia/jobs/5890112): $105,000 to $164,000

The rule of thumb is that employees cost the company double what they pay the employee. So, still hundreds of thousands per employee.

An engineer doesn’t cost $500k/yr. An engineer who lives in one of the highest cost-of-living places on earth costs $500k/yr. There’s absolutely no reason Wikimedia needs to pay that much.

> highest cost-of-living places on earth costs $500k/yr. There’s absolutely no reason Wikimedia needs to pay that much.

those places also have highest talent pool.

There’s more talented developers outside of high CoL places than inside. The comparison is not with one other place, the comparison is with the entire world.

It doesn't. Even with insurance, etc. I doubt they spend this much. They pay below market for most positions I've seen (though their benefits are on a good level) so even with taxes, overheads and all I don't see where 500k would come from.

Hire only the lowest cost employees, I demand it!

You’ve mildly annoyed me with that banner each of the thousands of times I’ve used your free world-library over these decades, and I’m done putting up with it!

> Hire only the lowest cost employees, I demand it!

Nobody said that. There’s a difference between “don’t hire the top 1% most expensive developers in the world” and “hire only the lowest cost employees”. Wikipedia can comfortably avoid the extremes on either end of the spectrum.

It would be nice if we had a "lot of lawyers", given how frequently we're sued to try and get content censored, or having to fight orders to hand over user data - and more generally, how massive these new laws we need to comply with are (see, e.g., the EU Digital Services Act, which even creates an entirely new annual independent audit process).

We even intervene in other court cases to try and prevent bad laws being created/interpreted in ways that would hurt the open internet (see, e.g., our amicus in the French Constitutional Court two weeks ago, our lawsuit against the US NSA, and our amicus briefs in the two US "Netchoice" US Supreme Court cases). We also operate the https://foundation.wikimedia.org/wiki/Legal:Legal_Fees_Assis...

Sadly, we're a very tight team. The downsides of being a nonprofit...

Anyhow, I'm going to assume people are just ignorant as to how much WMF does, not deliberately trying to undermine it. https://meta.wikimedia.org/wiki/Assume_good_faith , as they say.

(disclosure: lawyer for WMF)

It isn't a question of the good work you do.

People care about Wikipedia, not the Wikimedia Foundation. The criticism arises from misleading advertising. WMF fundraising conflates the two, implying that _Wikipedia_ needs money or it'll die. Meanwhile the 2023 budget shows $3.1m in hosting expenses versus $24.4m in awards and grants.

That is an issue. There is a number of projects that the Wikimedia Foundation want to do or be involved in, because they align with the mission. These all costs money, but are frequently of little interest to anyone not involved directly. There is absolutely no way to fund these, which leads to the foundation pushing for donations via Wikipedia, because that's the only thing enough people actually care about.

For the most part Wikimedia could kill off everything but English, Germany, French, Russian and a handful of other wikis and most people would be just as happy.

Wikimedia absolutely suck at telling people why they need the money. Technically the budget is completely transparent, it's just communicated extremely poorly.

Firstly, there's less conflation these days - go see recent banner wording for yourself. Secondly, if you're still just acknowledging Wikipedia hosting costs - and thus pretending there's (for example) no legal work necessary for it - I don't think people are getting through to you as they should. (And no, I'm not saying all legal work we do is a strict necessity for Wikipedia. Some is a strict necessity, and some is strategic e.g. an amicus, or the NSA lawsuit - but the latter does help secure a healthy environment for it and future projects that might want to take its place.)

> Secondly, if you're still just acknowledging Wikipedia hosting costs - and thus pretending there's (for example) no legal work necessary for it - I don't think people are getting through to you as they should.

From your phrasing (still) it seems like you might've confused me with the person you initially replied to.

I was comparing technical infrastructure costs to award/grant costs because most critics are going to view the former as essential and the latter as mission creep. I don't have any insight, nor do I have any inclination to criticize, your payroll.

> I was comparing technical infrastructure costs to award/grant costs

No, you were comparing a small part of technical infrastructure costs to grant cost.

Is every dollar spent mission critical to running wikipedia? Obviously not. But that doesn't mean its runnable on 3 million dollars.

> No, you were comparing a small part of technical infrastructure costs to grant cost.

I have no control over how WMF presents its expenses.

For years WMF foundation has run "we need money or Wikipedia will die" ads while spending a quarter of the budget on making grants. No one forced them to write that ad copy. It's progress that they've toned it down, but we shouldn't pretend that this criticism is surprising or completely unwarranted.

> I have no control over how WMF presents its expenses.

You have control over your reading comprehension. You called a number that was a very small portion of the technical infrastructure cost, the technical infrastructure cost.

You should also probably split out any of the grants related to technical infrastructure (i presume at least some of this grant money might have historically gone to wikimedia Deutschland to do technical infrastructure on wikidata, but im not sure off the top of my head)

I'm sure you could make many arguments that some of WMF's expenditures are not needed (i'd even agree). That doesn't mean it can survive on a few million dollars.

So you are saying that if people thought they were contributing to keep Wikipedia running because that is what the ads claimed, its their fault for not going through the financial reports to see where the money is going.

If you raise money saying it is for wikipedia, it should be spent only on wikipedia or IMO it is misleading.

Even "spent only on wikipedia" is a bit complicated -- bawolff's example was grants to Wikimedia Deutschland for work on wikidata, which sounds like it's some separate project. But really wikidata is used pretty extensively inside wikipedia, particularly for keeping facts synched up between the various project languages. Or money spent on Wikimedia Commons sounds like another random project, but actually it's the infrastructure for all the images you see on wikipedia.

It gets fuzzier as you go out to the promotion-of-free-knowledge stuff, for sure. You can argue its connection to keeping information being contributed to wikipedia, and the long term health of the community, but it's definitely less directly keep-the-lights-on.

Harvard is also exceptionally greedy to charge tuition, but also it wouldn’t be fair for legacy admits to get the brand and network without paying what is peanuts to most of those families.

LLM trainers need to pay for Wikipedia to help balance the information economy. If Google had to pay something substantial versus just lifting the content into their own “smart” results, then other sites would follow and wouldn’t have to rely on crappy SEO tricks.

Another perspective is that LLM trainers have largely been so disrespectful of IP / copyrights from their own greed that the content creators need to fight back with greed of their own. If the WM approach to media loses out to corporate and/or state control, it could e.g. make the Western internet much more like state-owned China.

Not entirely convincing arguments but it’s probably going too far to call WMF too greedy.

This is a rediculous thing to say. You think the interest off 250 million would be enough to run the wikipedia website?

Do you have any experience operating web properties at scale to come to this conclusion? Did you base this on anything at all?

I don't agree that it's pure greed but hosting costs for the Wikimedia Foundation in the FY2022-2023 were $3.1 million. [1][2]

[1] https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...

[2] https://wikimediafoundation.org/wp-content/uploads/2023/11/W... (page 4, pdf page 6 for expenses)

Sure, but hosting costs are a very small part of the costs of running a website (and tbh, its super unclear what hosting costs do and do not include. Do they include the cost of servers? Is it just peering costs and renting rackspace? At the very least they do not include staff. Servers are pretty useless if you don't have anyone to hook them up and manage them)

Hosting costs are £3m, but total expenditure is $160m - which obviously isn't covered by the interest on $250m.

I don't know what it takes at the scale of something like Wikipedia, but at my own scale, a single/self-employed developer, yes, hosting costs are a small part, but then I have to eat, a place to live, replace hardware, pay bills and taxes and much much more, things none of my sites could exist without.

Hosting is nothing without coders and maintainers, regardless of if the main content is contributed free, and decent coders and maintainers, software and hardware engineers, aren't cheap and don't count as hosting costs.

the expensive bit of serving big websites is the ads, the tracking, the analytics, the vast internal teams focused on endless avb testing etc. if you boil Wikipedia down it's mostly static pages with a crud editor. it's cached to the moon and back, most pages don't change. the pages are tiny. they're not paying aws for bandwidth.

> most pages don't change

Generally the pages that are viewed a lot change a lot. Sure, there is a long tail of mostly static pages, but that is not super relavent.

Hosting 80GB of data? Absolutely. Even if it was 800GB.

Umm, its roughly 500 terabytes if you include uploaded files, but that is besides the point.

Hosting a bunch of static data is really easy but only a small part of running a site like wikipedia.

1M$/year used to be enough.

Source: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...

Why stop there, the site cost $0/year back in 1999.

The growth of spending does not reflect the growth of usage or content, not even close.

Your own link shows investment income as about $3-4m. Nowhere near enough to pay for infrastructure and engineers, let alone all the other things you need to operate Wikipedia.

If you want to argue against finding Wikipedia you are going to have to do so in something approximating reality. Because the idea that they could somehow operate on one percent of their first budget is completely off base.

I think commercial offerings are good to offer as an alternative to donations for institutions that want to support something and cannot easily make a donations.

WMF goes the way of bureaucracies. It finds multiple ways to enlarge itself over the years and just cannot ever get any smaller. I suppose at some point it will overextend itself and harm Wikipedia, the only[1] thing of value, in the process.

[1] Yeah there are Wikidata and some other projects. But it's like saying Google is not an ad company.

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

for now

Wikimedia is Wikipedia's parent entity, but it's also the parent of:

    MediaWiki
    Wikibooks
    Wikidata
    Wikifunctions
    Wikimedia Commons
    Wikinews
    Wikiquote
    Wikisource
    Wikispecies
    Wikiversity
    Wikivoyage
    Wiktionary

https://en.wikipedia.org/wiki/Wikimedia_Foundation

It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.

Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.

Wikivoyage is underrated and that was not helped by the acrimonious split with Wikitravel (which was acquired a predatory marketing company), but it finally seems to be pulling ahead.

One of my favorite LLM applications is getting them to write wikidata queries. The data is amazing, but the query language is nothing but pure hell.

Disagree, this is too dismissive. Commons, Wikidata, and Wiktionary are all useful. Especially Wiktionary; probably one of the best online dictionaries imo. Often has a lot of unique info that's hard to find even in dictionaries, very good etymologies. All useful in ML.

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project.

I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. And it would be bad if didn't do this because it's a revenue stream for them.

There’s also Wikidata which has machine readable data for everyone

https://m.wikidata.org/wiki/Wikidata:Main_Page

> Wikidata is a free and open knowledge base that can be read and edited by both humans and machines.

> Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.

Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata

Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint.

Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/

DBPedia has been providing machine-readable info drawn from Wikipedia infoboxes for a long time, and that's absolutely an open project. But there are drawbacks to that approach, nowadays Wikipedia users are broadly expected to input that info into Wikidata (a language-independent project, providing its data under the CC0 license) and the Wikipedias are gradually moving towards fetching the information from there as well. The English Wikipedia is somewhat behind on that, so DBPedia might still be useful for parsing info from it.

Machine-readable infoboxes are surprisingly painful because an "infobox" isn't a top-level thing in mediawiki. Rather, it's something that emerged from community practice -- a template that's slightly-standardized, and has a sprawling network of subtemplates for specific categories of article. Then the assorted other language wikipedias have their own variants of these templates with their own names, etc.

Pulling it into a mediawiki extension (or core) and making it part of the page-level metadata gets suggested pretty frequently, but it's a bit contentious amongst the hardcore editors who'd need to actually adopt such things. The template-based nature of the current infoboxes mean that they're very accessible to the community, and it's easy to spin off new variants or make changes without getting programmers to help you.

There's slow movement towards getting the sort of data that winds up in infoboxes into wikidata, but it's still somewhat spotty.

(If you've never done it, it can be quite edifying to install mediawiki for yourself and seeing how much of the surrounding infrastructure of wikipedia is absent because it's all templates.)

I once tried to export a simple infobox template with dependencies and it exported 30-40 different template/module pages.

CC-BY-SA to be specific. How does attribution of derivative works work in LLMs (etc)? When is the content they produce required to be CC-BY-SA as well (by virtue of the -SA part of the license)?

LLM output can't be copywritten as it's not the work of a human. The company would presumably have to include attribution to every piece of data the system is trained on for every query? Seems absurd, but then how would attribution work? Not really considered when the license was written.

Licenses go beyond copyright though, they tell you what you can and can't do with something someone else made. They're contracts.

I'm sure we'll be seeing more lawsuits (and likely new regulation and licenses) around this.

Are you saying LLMs are not derivatives of their source material? Why not?

I agree on lawsuits being likely and needed to establish precedent. CC-BY-SA specifies "other rights such as publicity, privacy, or moral rights may limit how you use the material".

At some point, maybe poorly-trained chatbots that consistently produce what's seen as avoidably/negligantly poor results may become regulated. Like how if a company poorly trains its employees, it is on the hook for their employees' behavior.

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/article...

It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the United States is considered a fair use and does not require permission under copyright."

But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services, regardless of their works' legal status and Wikimedia's technical respect of that legal status.

> I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training

As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.

Given that all Wikipedia editors have explicitly consented to their content being released under the Creative Commons Attribution-ShareAlike 4.0 License, they don't get a choice about their content being used for any purpose.

Redistribution of content is an entirely different matter, and the legal status of copyrighted material in relation to LLM training is an open issue that is currently the subject of litigation.

Wikimedia Foundation’s perspective on this [1]:

> "it is important to note that Creative Commons licenses allow for free reproduction and reuse, so AI programs like ChatGPT might copy text from a Wikipedia article or an image from Wikimedia Commons. However, it is not clear yet whether massively copying content from these sources may result in a violation of the Creative Commons license if attribution is not granted. Overall, it is more likely than not if current precedent holds that training systems on copyrighted data will be covered by fair use in the United States, but there is significant uncertainty at time of writing."

The new Wikimedia Enterprise APIs facilitate attribution. For example, the "api.enterprise.wikimedia.com/v2/structured-contents/{name}" response [2] includes an "editor" object in a "version" object. So the Wikipedia editor who most recently edited the article seems quite feasible to attribute. ML apps could incorporate such attribution in their offering, and help satisfy the "BY" clause in the underlying CC-BY-SA 4.0 license for Wikipedia content.

---

1. https://meta.wikimedia.org/wiki/Wikilegal/Copyright_Analysis...

2. https://enterprise.wikimedia.com/docs/on-demand/#article-str...

As another editor, I think they might be a vocal minority. :)

> But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services,

I think that will heavily depend on just what the money goes to.

A better user experience, tightening up the code behind things, fewer nag screens for donations? Justifiable.

Jimmy Wales going from decently-compensated to a bad case of Founder Syndrome? Not so justifiable.

Founder Syndrome: a psychological condition wherein a person who starts a venture believes the venture should earn them a net worth on the order of billions of dollars, regardless of its actual economic value, and is willing to seriously enshittify the venture's product or service in order to make this delusion come true, especially in preparation for an IPO. See also: /u/spez, SPAC, Facebook, Unity Engine

> Jimmy Wales going from decently-compensated to a bad case of Founder Syndrome? Not so justifiable.

Jimmy wales is not paid at all (he has a board seat, but that doesn't come with any money)

He of course has leveraged his fame from being "founder" quite extensively. I think most of his money comes from fandom which he is also one of the founders of.

> machine parsable formats,

Isn't the point of wikidata to offer machine-parseable formats?

Also: how can those formats be locked into a paywall?

wikidata is not the same as wikipedia infoboxes.

wikipedia infoboxes API is already paywalled, pricing starts from $0.01/req.

What I am proposing is that Wikipedia infoboxes should be included in Wikidata

this is good proposal, but it is far from what we have and not clear if it will happen.

But now unfortunately the Wikimedia foundation has a motive to block such a proposal, in order to not threaten their revenue stream

And that's the problem with making money off convenience. If the free thing becomes better, then it cannibalizes the paid stuff. So if you take this approach to funding, you will want the free tier to be perpetually bad as much as possible.

I think sorting current infoboxes mess is not trivial problem. We will see what will be quality of their solution, it could be lower than actual wikidata.

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

Right, the incentives are adverse: if they lower quality of service for free offerings, they'll be rewarded with more paying customers for reliable service.

But quality does take money.

What are the incentives for more paying customers? Wikimedia Foundation, being a non-profit, has a lot less incentives to chase money at the expense of product quality.

The for profit parts are meant to chase money, and higher revenues justify higher management salaries ("I am running a bigger organisation so I deserve to be paid more"). This is widely accepted as being a motive for misaligned incentives in for profit businesses (with regard to takeovers) too.

> I'm running a mirror for OpenStreetMap

Curious why you're running a mirror for OSM, is it public or for your private use? If you don't mind, I'd love to learn more about it; sounds super interesting to me.

Wondering then if you've tried the Snapshot API in wikimedia enterprise?

Having the data available for download and a paywalled API is pretty standard practice for a lot of this kind of data. Copernicus weather/satellite data is often setup like this. There's a free download option, a free (limited) API, and then if you wanna get fancy you can use a more feature rich paid API. Many other weather services do similar. It seems to work pretty well with the usual caveats.

see you in a year when that finishes

It does exist, it is just incredibly painful to use -- it's a raw SQL dump of each individual page, which is missing some of the dynamically generated parts of the page you might care about (like categories, unless you separately download that table too). You also have to then run it through wiki software locally if you want the rendered page instead of markup.

Don't get me wrong, it is wonderful that the Wikipedia team offers this, and I am grateful they give anything at all for offline usage. It just feels like it's intended more as a side product of their backup process, rather than something you're really supposed to use.

What's up with HN's general tone of hostility toward Wikimedia foundation? It's weird how the more free a service is the more entitled our attitudes seem to become.

I think the hostility comes from Wikipedia constantly asking for money when most of the money goes to things that aren't Wikipedia.

Hiding behind a throwaway account allows you to make false statements, unless you can substantiate the "most of the money" comment. Wikimedia Foundation breaks down its expenditure here:

https://wikimediafoundation.org/support/where-your-money-goe...

I'm sure what poster is referring to was "discussed" previously in the link below. I use parentheses because it descended into flames pretty quickly:

https://news.ycombinator.com/item?id=33170710

My account is two years old and your profile is empty so we're equally as anonymous.

That aside, the first heading in your link is "43% Direct support to websites" so yes, most of the money does not go to Wikipedia.

> Real-time access to Knowledge

Wikipedia knowledge is basically "democracy" knowledge, i.e. the more people decided to support an idea, the "truer" it gets. That's not knowledge at all!

That politicians get to scrub their pages shows there are cracks in places, but overall it's generally pretty ok

> overall it's generally pretty ok

That's the kind of glowing praise I'd get from my 8th grade Geometry teacher when I got a C on a test.

lol, I find wikipedia to be a bit all over the place or lack structure. It also is largely a giant block of text, though I do think the new LHS menu has improved things

That's exactly what almost all knowledge is.

When was it that you last verified something by yourself, with an experiment?

You didn't test the things you know. You know things because you could see they were the consensus, and so you had no reason to challenge them.

If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unreliable people. So with the Wiki.

Human knowledge isn't from the platonic realm. Human knowledge isn't checked by a theorem prover. You get almost all of your knowledge from other people, and you have no choice but to trust them for almost all of it.

“That's exactly what almost all knowledge is.”

It’s worse than that. Much knowledge comes from authorities or peers. The sources could be unpopular or barely reviewed. Yet, people are likely to believe specific types of sources. It seems to be hardwired for some purpose.

We trust the consensus of published, peer-reviewed experts. That's different than the kind of Demos that does things like declare war on Persia, kill Jesus and Socrates, or edit Wikipedia.

> If an idea is disputed, then you trust it less. If it comes from a small number of reputable sources, then you trust it more than a large numbers of unreliable people. So with the Wiki.

Right. That makes Wiki kind of unreliable. Not completely. And not to the point of uselessness, but you should trust it about as far as you can throw it.

> Human knowledge isn't from the platonic realm.

Citation needed ;)

We humans also trust the consensus of non-peer reviewed truth all the time. Tell that group of children that the opposite sex doesn't have cooties, and there's a good chance they'll laugh at you. Look at any online community, and it's the same. We humans are great at it and do it all the time.

>We trust the consensus of published, peer-reviewed experts. That's different than the kind of Demos that does things like declare war on Persia, kill Jesus and Socrates, or edit Wikipedia.

Ye of little faith! The Demos, after much bickering, have also decided to largely trust the consensus of published, peer-reviewed experts.

But what you really have to ask when you say you trust the consensus, is who forms the consensus of peer-reviewed experts?

It's rare to get an explicit consensus from an actual organization. When Cochrane does a large meta-analysis, and whisper "moderate evidence", I stop reading and immediately trust them with my life. Unfortunately, they very rarely have confidence in anything.

Most of the time, the consensus of published, peer-reviewed expert is also not something people form on their own. When has your neighbor last read and synthesized the literature to determine what the consensus is on hydroxyapatite in toothpaste, before going to the store?

Individual experts, I also trust only as far as I can throw them. The consensus of experts I'm happy to rely on. But that, very often, also comes from trusting the Demos, I'm afraid.

When was it that you last verified something by yourself, with an experiment?

Just a few weeks ago. I did a simple experiment to check whether the Super-94 at my local Chevron is indeed ethanol-free. It wasn't.

Now imagine you were able to go and edit the sign that says it is ethanol free to add the details of your test and dispute the claim, that would improve the knowledge.

For that to make it into Wikipedia you'd have to first write an article in a reputable source.

Or rather, have a reputable source write an article about your work. If you write the news article about your own work and they publish it, the article is still a primary source (despite not being self-published)!

This is nice analogy, but a wrong analogy. Wikipedia specifically does not allow original research.

https://en.wikipedia.org/wiki/Wikipedia:No_original_research

I thought knowledge, at least the best type comes from primary sources and from repeatable experiments with explicit premises as much as possible. This makes it sound like all knowledge is hearsay. If it is, what is the point of a place like Wikipedia or even an encyclopedia?

Indeed, the best type of knowledge comes from primary sources and original research. But those also produce an awful lot of not-knowledge.

Wikipedia's approach to sifting the knowledge from the not-knowledge is to prefer reliable secondary sources, i.e. sources deemed capable of telling the difference, mainly because they have a reputation for good editorial control. It's far from an ideal touchstone; but relying on "experts" is worse, because who's an expert? You need experts to identify experts, which is circular.

"Reliable secondary sources" doesn't amount to hearsay.

>That's exactly what almost all knowledge is.

That's what almost all human assumption and belief system is, also ideology and religion, but knowledge is indeed something different, and not of a type that should rely on democratic consensus. It instead needs to be held up by material evidence that's always subject to retesting no matter how unpopular a new idea is. This is obvious.

The rest of what you say could just as easily be applied to the foolish social dogmas of nearly any past age in human history, dogmas that so often turned out to be wrong. A small number of reputable sources (for their time) upheld doctrines such as geocentrism, religious extremism, hatred for certain racial groups and numerous fervent beliefs in the right of certain people to dominate others. These are just a few examples.

A more material one would be the certainty among reputable sources that plate tectonics were nonsense, until of course they were shown not to be by what started as an argument by only a few people who were deemed very unreliable.

None of this is to give weight to every crackpot idea put forth, or claim that all opinions are equally valid until stated otherwise, but what makes the difference is evidence, not consensus.

If I'm interpreting your comment correctly, I don't think your presentation of the role of popular consensus on Wikipedia is accurate. Read this page: https://en.wikipedia.org/wiki/Wikipedia:Consensus.

Wikipedia doesn't establish consensus by pure numbers or voting, although it is a contributing factor. In disputes, it has moderated discussions with verdicts given by elevated users, including admins. Things like statistics and even (perhaps especially) precedent all weigh in. Popularity of a side can be weighted, but ruling purely based on popularity is actively discouraged.

This can lead to scenarios where 90% of users want something, but the moderater rules along with the 10%. Often, this happens when the discussion was initially among a bunch of relatively new users who aren't aware of some policy, and a more experienced editor points that a dispute is clearly not in line with some policy. This happens very regularly and is often a source of drama with long discussions.

This process actually arguably works better on popular and contentious pages; you get eyes and discussions of substance on those. Most boring pages are virtually ghost towns and are counterintuitively more susceptable to popularity-based consensus. Whatever you put up will likely stick, so it's just a matter of how many people and who will protect the page for the longest.

Also read this page: https://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_vie....

The second page addresses your concern about not giving too much weight to fringe theories. It's not enforced as well as it could be in many places though; it can be hard to judge what's due or undue weight.

Read up on Wikipedia's "reliable source" policies.

Information on Wikipedia is meant to be backed up by a verifiable source, partly to prevent a situation where knowledge only makes it onto Wikipedia if enough of the editors agree that it should be true.

Molly White made a great video and write-up explaining this a few months ago: https://blog.mollywhite.net/become-a-wikipedian-transcript/#...

I know about that, but it's basically the same thing since reliable sources in the wikipedia terms are a set of sources that we collectively decide to trust. What's interesting about wikipedia sources is that it won't allow you to directly quote a person even though they are a well known trustworthy information source. Instead you must back up your statements through a 3rd party (usually media-related) entity. This is both good and bad, considering that journalists may not be the best at evaluating certain information, especially in the science or politics field.

>it won't allow you to directly quote a person even though they are a well known trustworthy information source

Little known fact: That is actually allowed in some limited situations, but only reluctantly, and with a lot of care.

For instance if someone is recognized as an established researcher in their field with publications in top academic journals and then they make a statement on their website about something they have expertise in, you can actually cite that if you have no better source! Even though it's a direct self-published quote.

... and Wikipedia does not consider itself a reliable source. Rightfully so. Open any politics-related article and you'll see why.

Nitpick, the purple blue gradient is overused, especially as the background of source code examples. It hinders readability and feels off-brand for a Wikimedia project. This might be foreshadowing a descent into chasing trends. I just hope Wikimedia does not follow Mozilla's track in spending budget on fluff work.

I was hoping more groups did stuff like this. The free sites doing it could handle some copyright issues if their EULA had a built-in license for distribution.

In my previous analysis, (IIRC) I found that Wikipedia articles were under a copyleft license with attribution requirements. Does how Wikipedia Enterprise delivers this bypass that where neither use nor derivatives have those requirements? Or are they ignoring the licensing requirements which still legally apply to enterprise customers? Or are they not even claiming to address it?

My understanding is that Enterprise is mostly about smoother delivery of content. E.g. Google has those informational summaries in search results, and it's easier for it to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out. It also puts it all into conveniently formatted responses, and tries to do some signaling of content reliability.

The API responses do include information about the content license, which differs a bit between different wikimedia properties: https://helpcenter.enterprise.wikimedia.com/hc/en-us/article...

(I work for the WMF. I don't work on the Enterprise stuff, or have any insider knowledge of it. E.g. I used Google as an illustrative example, but I have no idea whether they're actually using this service. :D)

> to keep those current if there's a stream of updates it can subscribe to, rather than having to constantly download the full Wikipedia database dumps and parse them out.

I mean presumably prior to this they were using the [free] parsoid rest api along with the [free] event stream api. I highly doubt they were parsing the dumps.

Its not even clear to me what the core value proposition of the new api is over the old api from the perspective of google. Maybe SLAs? A way to justify donating money without it looking like a donation?

No idea whether it's any quicker than the event stream APIs, but it sounds like the machine-readable aspect does a lot of processing that just using parsoid wouldn't get you. Parsoid is only a wikitext-to-HTML service, so you'd still need to do significant work on it to get meaning out of that HTML. The data dictionary page[1] suggests that it's exposing things like "how many citation-needed templates are in this?" (presumably localized, since those template names change across wikis), which I could see being handy for services like that.

That said, I suspect that the real value is just the "reliable" bit on that summary page.

> Written agreements, no additional content restrictions, SLAs, 24/7 support, and contractual guarantees against surprise breaking changes make working with Wikimedia Enterprise safe and reliable at scale.

i.e. If I was building a big enterprise thing using wikipedia data, getting a SLA and a promise that us rando open source devs won't completely change the APIs on a whim would be nice.

[1]: https://enterprise.wikimedia.com/docs/data-dictionary/

There's also r&d going into additional, inferred data layers, e.g.: https://m.mediawiki.org/wiki/Wikimedia_Enterprise/Breaking_n...

All licensing requirements still apply.

However keep in mind that different wikimedia content is under different licenses (e.g. wikidata is public domain). Additionally in the united states, if you take just the factual content (removed from any creative context) the facts are not copyrightable so the license does not apply.

I'm reading the API docs https://enterprise.wikimedia.com/docs/

And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??!

Seriously it's 2024! Not having a spec to download and insepct for any API is a sign of incompetence. When tools like Postman or https://github.com/OpenAPITools/openapi-generator-cli exist and save hours of time, you can't seriously expect devs to write all this connecting code by hand anymore.

If only there was some code generation technology that could take in human-readable specifications, not just machine-readable ones....

Then create one, share it and save us some time.

so enterprise very wow

link at bottom of https://enterprise.wikimedia.com/docs/ page

Bottom of the page ...

https://api.enterprise.wikimedia.com/spec/spec.yaml

> And they don't have an OpenAPI spec available to download? So they seriously expect developers to just manually write their own client code by reading and translating those example CURL commands by hand??!

This is baby work to literally every programmer I know.

Not to mention that if it's really needed, the LLMs this is made for also excel at converting CURL commands to whatever spec pretty trivially. I've used them this way on a few occasions.

It's also a solved problem that literally no human should have to bother with anymore. Like it was solved with SOAP APIs back in the 2000's.

Not to mention the fact that documentation is still always hand-written - which can have mistakes or typos in it, then downstream developers will have to translate that into hand-written client code, which can also have those same mistakes or bugs in it. And when the mistake is corrected, the dev needs to go back to where the correction was made in the docs, re-read and interpret the correction and then correct their client code. With API specs and code generators that entire process is automated now.

Now imagine bringing together approximately 6 different APIs into a single app, 5 have API specs while 1 doesn't. Now 80% of your time is spent on that 1 API where you have to read the docs to keep your client code up to date. Oh that 1 API vendor doesn't bother letting you know when their API is updated! So you have to F5 their website every week! This is literally my life right now.

Anyway, in this case, it was my mistake not to read until the very bottom where they had an OpenAPI yaml spec link there.

> It's also a solved problem that literally no human should have to bother with anymore.

This is such a hilariously wrong statement I can't even begin to approach the rest of this comment with any serious attention.

If someone from the wikimedia enterprise team is here and interested in providing those SDKs, but finds the openapi generator linked above to be lacking, I'd be happy to help (my company does the SDKs for OpenAI, Cloudflare, etc).

I've been a monthly donor to wikimedia for a long time and it'd be quite a privilege to support them with nice API wrappers too. Email in profile.

Wikimedia is unfortunately becoming one of the worst places to give your money to. They have their closed-source infrastructure, which now for years has not been able to generate HTML dumps without a significant percentage of articles missing.

They have know of the bug for ages, but still...

WMF don't care about their products at all, they give much more funding to vanity workshops in Africa while utterly ignoring the requests of the Wiktionary community, for example.

Interesting, the current DB dump is not for everyone, but if they also offer an LLM trained on Wikipedia data that answers and provide actual valida citation , please do. (not sure if duckduckgo stop offering that)

AFAIK a good way to provide better answers and avoid hallucinations would be to compute embeddings for all sections of text in Wikipedia and then when a user asks a question create an embedding from that question.

Use it to find the X closest embeddings to the question being posed, lookup their original articles, feed them all into context of an LLM and then ask it to answer the question based on that context (alone).

Contexts are becomming quite large so it's possible to put a lot of stuff in there. LLMs answering questions based on a giben text seem to be more reliable than those that are simply trained/fine tuned on some library of texts. p

Unfortunately this still results in plenty of hallucinations.

What do you mean? Have you or someone else already followed this exact approach?

The approach described above is what is commonly referred to as RAG[0]. I am not aware of someone having used it on Wikipedia but, from experience and while it helps, it does not eliminate all hallucinations.

[0]: https://en.wikipedia.org/wiki/Large_language_model#:~:text=t...

Supabase did: https://supabase.com/blog/chatgpt-supabase-docs

I've attempted it as well a year ago (mostly for fun) for our project.

Yes, it can still hallucinate. But I would say it's much much much better in this regard than fine-tuning.

When I did it, the main issue was that our documentation wasn't exhaustive enough. There are plenty of things that are clear to our users (other teams in the company), but not at all clear to the LLM from the few text excerpts it receives. Also, our context was quite limited back then to just a few paragraphs of text.

You can do this with the copilot chat feature in MS Edge. I just tried to ask it to use only wikipedia and it gave me four references, two of which were wiki. So at least you can get it to spit out references with a bias

Not quite what you're looking for, but something to play around and give feedback on in this area is https://meta.m.wikimedia.org/wiki/Future_Audiences/Experimen...

I support this and think it should be a model for other organizations. They already publish the data for free. It's already being used to train models. Companies are already using those models to profit. I'm not sure I understand the concern.

I hope to see their APIs expand in future work, successfully tying together machine learning with their prior work on publishing a knowledge graph enriched with high quality structured metadata.

Is it’s ok to train AIs on CC-licensed data? What about attribution, should the AI be distributed with a note about the dataset’s origin?

I could imagine a poorly trained model that returns content nearly identical to original; who would enforce CC rules then?

It shouldn't be much of a problem to ship the LLM along with attributions. (List of all sources used in the dataset - not a problem, unless they are secret, shady or illegal.) Wikipedia is one of the easy ones, since you need to attribute just 'Wikipedia' for the entire corpus instead of many individual users.

The bigger issue is when a user uses the model to generate some text. Should they attribute it when using it somewhere?

That doesn't seem very practical, since it seems that soon most of the text will be edited by LLMs and those seem to be trained on most of the web -> so pretty much everything would need to be attributed to everything. Unless someone puts a stop to this, which I find improbable.

Including a list of every Wikipedia author is possible but very inconvenient. https://en.m.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_...

I wonder if it can be like those California proposition notices, just include it everywhere just in case

https://creativecommons.org/2023/08/18/understanding-cc-lice...

I give it at most a year before they start using this LLM to generate content for Wikipedia

Good so they can stop shilling for donations on wikipedia

Looking forward to seeing the details on how they will handle revenue sharing with all of the people who contribute to them.

are they obligated?..

No, but nobody is obligated to contribute to or support them either.

that's not really true. Because of network capture, those who wants to contribute into some public free human knowledge, obligated to do it through wikimedia and contribute to wikimedia revenue streams.

Or this simply discourages contributions for folks not wanting to volunteer effort to the gaping maw of yet another monolithic content vacuum/LLM.

If you want monetary compensation you should not be contributing to Wikipedia. Otherwise apply to work there as an employee.

After seeing this comment, onlookers report that the Wikimedia Foundation appears to have just vanished in a puff of volunteerism

The Wikimedia Foundation are paid employees.

They sure are!

I think the parent comment to mine was edited, and now my reply looks weird. This is what I remember replying to:

"If you want monetary compensation you should not be contributing to Wikipedia"

Probably the same ways as billion dollar businesses share revenue with open source developers...

Funding that goes to editors (for example, stipends to travel to Wikipedia conferences) has been decreasing steadily each year. It's not due to a lack of money - Wikimedia consistently brings in millions of surplus revenue each year, see WP:CANCER[1] - so it's not clear that an additional revenue source will change anything.

^1: https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C...

The details are: none. There is no revenue sharing. That's literally the point of the Creative Commons license that Wikipedia uses. There's some things on the margins about attribution strings and share-alike requirements, but none of it would render AI training on Wikipedia illegal or compel AI companies to seek a separate, royalty-bearing license. Creative Commons is a "do what you wish" license, not a "free until I want money, then I rugpull you" license.

If you wanted revenue sharing, you were at the wrong party. You wanted the Microsoft-sponsored "Open Source is Communism" party down the block.

Reply on news.ycombinator.com