Hacker News Reader

Comment by mushufasa | Hacker News Reader

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project.

I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. And it would be bad if didn't do this because it's a revenue stream for them.

There’s also Wikidata which has machine readable data for everyone

https://m.wikidata.org/wiki/Wikidata:Main_Page

> Wikidata is a free and open knowledge base that can be read and edited by both humans and machines.

> Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.

Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata

Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint.

Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/

DBPedia has been providing machine-readable info drawn from Wikipedia infoboxes for a long time, and that's absolutely an open project. But there are drawbacks to that approach, nowadays Wikipedia users are broadly expected to input that info into Wikidata (a language-independent project, providing its data under the CC0 license) and the Wikipedias are gradually moving towards fetching the information from there as well. The English Wikipedia is somewhat behind on that, so DBPedia might still be useful for parsing info from it.

Machine-readable infoboxes are surprisingly painful because an "infobox" isn't a top-level thing in mediawiki. Rather, it's something that emerged from community practice -- a template that's slightly-standardized, and has a sprawling network of subtemplates for specific categories of article. Then the assorted other language wikipedias have their own variants of these templates with their own names, etc.

Pulling it into a mediawiki extension (or core) and making it part of the page-level metadata gets suggested pretty frequently, but it's a bit contentious amongst the hardcore editors who'd need to actually adopt such things. The template-based nature of the current infoboxes mean that they're very accessible to the community, and it's easy to spin off new variants or make changes without getting programmers to help you.

There's slow movement towards getting the sort of data that winds up in infoboxes into wikidata, but it's still somewhat spotty.

(If you've never done it, it can be quite edifying to install mediawiki for yourself and seeing how much of the surrounding infrastructure of wikipedia is absent because it's all templates.)

I once tried to export a simple infobox template with dependencies and it exported 30-40 different template/module pages.

CC-BY-SA to be specific. How does attribution of derivative works work in LLMs (etc)? When is the content they produce required to be CC-BY-SA as well (by virtue of the -SA part of the license)?

LLM output can't be copywritten as it's not the work of a human. The company would presumably have to include attribution to every piece of data the system is trained on for every query? Seems absurd, but then how would attribution work? Not really considered when the license was written.

Licenses go beyond copyright though, they tell you what you can and can't do with something someone else made. They're contracts.

I'm sure we'll be seeing more lawsuits (and likely new regulation and licenses) around this.

Are you saying LLMs are not derivatives of their source material? Why not?

I agree on lawsuits being likely and needed to establish precedent. CC-BY-SA specifies "other rights such as publicity, privacy, or moral rights may limit how you use the material".

At some point, maybe poorly-trained chatbots that consistently produce what's seen as avoidably/negligantly poor results may become regulated. Like how if a company poorly trains its employees, it is on the hook for their employees' behavior.

It also seems Wikimedia isn't trying to relicense the content in any way that strips its e.g. CC-SA status, but rather providing the licenses as context alongside each API call. https://helpcenter.enterprise.wikimedia.com/hc/en-us/article...

It's worth noting that https://creativecommons.org/faq/#artificial-intelligence-and... itself takes the general stance that "as a general matter text and data mining in the United States is considered a fair use and does not require permission under copyright."

But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services, regardless of their works' legal status and Wikimedia's technical respect of that legal status.

> I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training

As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.

Given that all Wikipedia editors have explicitly consented to their content being released under the Creative Commons Attribution-ShareAlike 4.0 License, they don't get a choice about their content being used for any purpose.

Redistribution of content is an entirely different matter, and the legal status of copyrighted material in relation to LLM training is an open issue that is currently the subject of litigation.

Wikimedia Foundation’s perspective on this [1]:

> "it is important to note that Creative Commons licenses allow for free reproduction and reuse, so AI programs like ChatGPT might copy text from a Wikipedia article or an image from Wikimedia Commons. However, it is not clear yet whether massively copying content from these sources may result in a violation of the Creative Commons license if attribution is not granted. Overall, it is more likely than not if current precedent holds that training systems on copyrighted data will be covered by fair use in the United States, but there is significant uncertainty at time of writing."

The new Wikimedia Enterprise APIs facilitate attribution. For example, the "api.enterprise.wikimedia.com/v2/structured-contents/{name}" response [2] includes an "editor" object in a "version" object. So the Wikipedia editor who most recently edited the article seems quite feasible to attribute. ML apps could incorporate such attribution in their offering, and help satisfy the "BY" clause in the underlying CC-BY-SA 4.0 license for Wikipedia content.

---

1. https://meta.wikimedia.org/wiki/Wikilegal/Copyright_Analysis...

2. https://enterprise.wikimedia.com/docs/on-demand/#article-str...

As another editor, I think they might be a vocal minority. :)

> But as a practical matter, I wouldn't be surprised if some Wikipedia editors balk at their volunteer work being actively marketed and reformatted for ease of LLM training by the very platform that solicited their volunteer services,

I think that will heavily depend on just what the money goes to.

A better user experience, tightening up the code behind things, fewer nag screens for donations? Justifiable.

Jimmy Wales going from decently-compensated to a bad case of Founder Syndrome? Not so justifiable.

Founder Syndrome: a psychological condition wherein a person who starts a venture believes the venture should earn them a net worth on the order of billions of dollars, regardless of its actual economic value, and is willing to seriously enshittify the venture's product or service in order to make this delusion come true, especially in preparation for an IPO. See also: /u/spez, SPAC, Facebook, Unity Engine

> Jimmy Wales going from decently-compensated to a bad case of Founder Syndrome? Not so justifiable.

Jimmy wales is not paid at all (he has a board seat, but that doesn't come with any money)

He of course has leveraged his fame from being "founder" quite extensively. I think most of his money comes from fandom which he is also one of the founders of.

> machine parsable formats,

Isn't the point of wikidata to offer machine-parseable formats?

Also: how can those formats be locked into a paywall?

wikidata is not the same as wikipedia infoboxes.

wikipedia infoboxes API is already paywalled, pricing starts from $0.01/req.

What I am proposing is that Wikipedia infoboxes should be included in Wikidata

this is good proposal, but it is far from what we have and not clear if it will happen.

But now unfortunately the Wikimedia foundation has a motive to block such a proposal, in order to not threaten their revenue stream

And that's the problem with making money off convenience. If the free thing becomes better, then it cannibalizes the paid stuff. So if you take this approach to funding, you will want the free tier to be perpetually bad as much as possible.

I think sorting current infoboxes mess is not trivial problem. We will see what will be quality of their solution, it could be lower than actual wikidata.

Wikimedia is Wikipedia's parent entity, but it's also the parent of:

    MediaWiki
    Wikibooks
    Wikidata
    Wikifunctions
    Wikimedia Commons
    Wikinews
    Wikiquote
    Wikisource
    Wikispecies
    Wikiversity
    Wikivoyage
    Wiktionary

https://en.wikipedia.org/wiki/Wikimedia_Foundation

It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.

Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.

Wikivoyage is underrated and that was not helped by the acrimonious split with Wikitravel (which was acquired a predatory marketing company), but it finally seems to be pulling ahead.

One of my favorite LLM applications is getting them to write wikidata queries. The data is amazing, but the query language is nothing but pure hell.

Disagree, this is too dismissive. Commons, Wikidata, and Wiktionary are all useful. Especially Wiktionary; probably one of the best online dictionaries imo. Often has a lot of unique info that's hard to find even in dictionaries, very good etymologies. All useful in ML.

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

Right, the incentives are adverse: if they lower quality of service for free offerings, they'll be rewarded with more paying customers for reliable service.

But quality does take money.

What are the incentives for more paying customers? Wikimedia Foundation, being a non-profit, has a lot less incentives to chase money at the expense of product quality.

The for profit parts are meant to chase money, and higher revenues justify higher management salaries ("I am running a bigger organisation so I deserve to be paid more"). This is widely accepted as being a motive for misaligned incentives in for profit businesses (with regard to takeovers) too.

> I'm running a mirror for OpenStreetMap

Curious why you're running a mirror for OSM, is it public or for your private use? If you don't mind, I'd love to learn more about it; sounds super interesting to me.

Wondering then if you've tried the Snapshot API in wikimedia enterprise?

It does exist, it is just incredibly painful to use -- it's a raw SQL dump of each individual page, which is missing some of the dynamically generated parts of the page you might care about (like categories, unless you separately download that table too). You also have to then run it through wiki software locally if you want the rendered page instead of markup.

Don't get me wrong, it is wonderful that the Wikipedia team offers this, and I am grateful they give anything at all for offline usage. It just feels like it's intended more as a side product of their backup process, rather than something you're really supposed to use.

Having the data available for download and a paywalled API is pretty standard practice for a lot of this kind of data. Copernicus weather/satellite data is often setup like this. There's a free download option, a free (limited) API, and then if you wanna get fancy you can use a more feature rich paid API. Many other weather services do similar. It seems to work pretty well with the usual caveats.

see you in a year when that finishes

for now

Reply on news.ycombinator.com