AI tidies up Wikipedia's references and boosts reliability
nature.com
nature.com
My goal was to learn more and go deeper on subjects, but I was stunned by how often the linked citation didn't support the claim in the Wikipedia article. There were many times where the linked citation said the opposite of the Wikipedia article.
My theory was that overly competitive Wikipedia authors were skimming PubMed abstracts and assuming the paper would support their assertion. Ironically, some of the statements with 5 or more citations were the most incorrect.
Trying to correct these articles is some times like going to war with editors who refuse to admit they were wrong.
This is often the case outside of Wikipedia as well. Truthfully, I'm a layman because I don't have an advanced degree that suggests I'm qualified to interpret papers, but things are bad enough IMO that part of me wishes people would stop trying to communicate science to the public until this broken system heals at least somewhat. On a related note, it doesn't help that folks like Andrew Huberman are normalizing this idea that one or two studies are good enough to form conclusions about how average people can "optimize" (micromanage) their lives in ways that are clinically relevant. This isn't to take away from the good that Huberman does, but I think it send a the wrong message to people who don't actually have any experience reading papers.
One thing that might help is forbidding abstracts and conclusion sections in papers published in journals, which might cut down on some of the misunderstandings and make it harder to pass off a paper as support for a claim.
> Trying to correct these articles is some times like going to war with editors who refuse to admit they were wrong.
There is virtually no point in trying to fix Wikipedia pages until Wikipedia fixes themselves. They have an extremely hard job, but their prolific editors are too bully-like and contradict themselves all the time.
I think this all really the cost of admission for the ease of self-publishing. Now any jerk off can post about their opinions on keeping abstracts while using the bathroom on company time.
As you say there's a pretty big gap with the way researchers read through papers and the way the science media communicate about research with laypeople. It's a hard problem that needs more attention than it gets.
(Probably one of the most useful classes I took in grad school was a class in the beginning that forced us to read 10 papers a week and write summaries. The class was hell as obviously there was no way to properly understand each paper on top of your research and other classes. But it made us great at being able to quickly winnow papers for the meat of each paper and also made us great at finding papers with... dubious methods.)
I often will read something like "A connection between X and Y"
Then skim down to the conclusion and it will say "In conclusion, we were only able to see some weak connection and we aren't sure if it was one at all, but this warrants further research..."
I know negative results are still valuable science, but I feel like a good chunk of the papers I'm reading are basically worthless CV padding.
This happens in the HN comments all the time. People just assume the other person isn't actually going to stop read it.
Often they just read the wikipedia article that cited the paper.
Turns out its easy to find a reference for anything if you search long enough.
In other words, citations alone are insufficient
you just summed up wikipedia on any political issue.
My big problem and theory of what will be the downfall of Wikipedia: No Original Research.
WP is, unfortunately, not obviously any worse than researchers in general are: https://gwern.net/leprechaun#miscitation
> Trying to correct these articles is some times like going to war with editors who refuse to admit they were wrong.
Yep. Because now you are doing 'OR' by interpreting the paper, especially when the abstract is just lying/spin.
Now I don't really know how to use wikipedia but I thought it was one of those fake edits people might do as a joke. Went checking edit history and stuff to find out and turns out someone else tried to point out the myth too and some jackass with authority is jealously guarding that myth to keep it on the page forever.
Went and checked the government documents myself, among other things there is no Colt material making that claim in that context. Government also ordered cleaning kits, Colt supplied cleaning kits, so it's also contradicted by fact there too. Simple thing is there just weren't enough to go around, supply shortages, and someone at some point created the myth by misunderstanding that a design to "reduce fouling" thereby "reduce cleaning" don't in any way no hell no how imply "self-cleaning" in that way. Yet the claim remains on wikipedia, without clarification, because I guess some idiots repeating the myth in a book makes the myth okay.
I went and checked that book, too. No source of the myth in the cited book. It's completely fuckin made up and anyone with half a brain and a day using a firearm would know that, but there it is. All because someone, for some reason only God knows, personally wants it to be there and has the authority to keep it there.
Simple enough fix to my mind. If them "dragons", and I like that image there I think it fits, pull bullshit like that you just remove them. Which you can now easily do because whether or not Colt claimed something is a matter of truth that can be checked. In this case obviously not. Simplifies everything and gets people talking about the truth and how to best represent what's true, even if it's disputed, instead of having what's very clearly false with no hope of even clarifying "this appears to be a myth".
Wikipedia presently says:
> However, the rifle was initially delivered without adequate cleaning kits[43] or instructions because advertising from Colt asserted that the M16's materials made the weapon require little maintenance, and was capable of self-cleaning.[67]
The book, The M16 by Gordon L. Rottman, says on page 20:
> Most Marine units began receiving the XM16E1 in April 1967 and immediately experienced problems arising from several factors. Most units received little if any cleaning gear beyond some cleaning rods and bore brushes. Some units had never heard of chamber brushes. Colt is said to have hyped the weapon as futuristic, requiring little maintenance owing to new materials. This was interpreted to mean the black rifle was “self-cleaning.”
So Colt supposedly saying the rifle requiring "little maintenance" was then subsequently interpreted (by the Marines I think) to mean the rifle was "self-cleaning". The book doesn't say Colt made the "self-cleaning" claim, but whoever wrote that part on wikipedia is attributing the claim to Colt.
Hard to say if even the book's claim is right.. "Colt is said to have..." said by who? The book doesn't actually cite any Colt marketing material or anything like that.
Still on wikipedia though. Because writing bullshit without any source is fine as long as it's in print, I guess?
Does that means it's true? I highly doubt it. I wasn't alive in the 60s, but I've known a lot of Marines in my life. I was in the Army myself. I've known people who were Marines in Vietnam. I've never known anyone who ever operated a rifle who would interpret a claim this way or belief a self-cleaning rifle was a possible thing.
But unfortunately, even if you are personally an expert, Wikipedia doesn't let you come in and tell a page it's wrong. You have to publish your knowledge in an authoritative archival source and then it can be cited. I get that it can be frustrating as a subject-matter expert, but I don't know what better sourcing and citation rules an encyclopedia can have. They don't internally litigate the validity of a claim. They define what outside sources count as citable and then trust those sources.
Hell, I experiened a fairly stupid version of this a few months ago. I edited the page for Slayer's Reign in Blood in the section for pop culture references, adding a mention that Angel of Death was used in the Leftovers when Nora pays a prostitute to shoot her and uses the song to cover the gunshot sound. I linked to an episode summary on another wiki and that got deleted because apparently Wikipedia doesn't allow other wikis to be cited as sources. Fair enough as a general rule, though I think it doesn't make sense in this context because Wikipedia has episode summaries of television shows that don't cite sources at all. So I just removed the cite and linked to HBO's home page for the episode and that stuck, even though it doesn't have a description and you'd have to actually watch the episode to confirm I'm not lying.
But hey, whatever, rules are rules. No rules are perfect. Courts get it wrong sometimes, too.
seems like best thing would be to be able to say "this is not verifiable" or similar. seems to me these issues people have could be solved or improved a lot by having more and more nuanced info not less. Like if people are just so damn insistent the myth stay up there it should also be fine to point out "though there seems to be no evidence supporting this claim or claimed notion" when the books or stuff cited have nothing but more claims. More info, more context, not less
[0] https://scholarlykitchen.sspnet.org/2022/11/01/guest-post-wi...
[1] https://en.wikipedia.org/wiki/Impact_factor
I keep thinking of that quote from the Office: "Wikipedia is the best thing ever. Anyone in the world can write anything they want about any subject, so you know you are getting the best possible information."
wikipedia accepts published papers from journals as references, despite that those are primary sources. (I thought they originally didn't, but they do now)
How would this guy's theorems be on wikipedia if he hadn't published them? Wikipedia is leery of people editing "their own page" in the sense of conflict of interest, but cleary if you are correcting a published theorem, that is entirely within bounds. You could say on the Talk page "I published this theorem, wikipedia has it wrong."
What was the equation?
Yes, there might be bad actors, but it doesn't mean that everything is bad and impossible. You should try first, at least.
Previous comment of mine on this issue (including a bit about how to make URLs more specific): https://news.ycombinator.com/item?id=23897686
No one could find a reference to contest it with, because nobody was writing articles defining terms that everyone understood; and the original editor's reference was not available online at the time, so no one could look at it and say "Uh, this is just some rando saying it would be cool if we called secondary-world stories 'high fantasy'."
Now the original, bogus reference is gone from the article, replaced with 20 low-effort articles that are obviously sourced from the same Wikipedia page they're being used to support. Boom, citogenesis! https://xkcd.com/978/
There is nothing thing ironic about it. Wikipedia is run by bullies who make up references.
With minimal effort, humans hookup AI to do some job, and things "just get better" rather than entropy taking its natural course and many things (without maintenance) trending towards "worse".
Once you have a bunch of this human / super-human level doing mundane things on wikipedia, they're now there in perpetuity, constantly improving.
I suspect this is what's going to start to happen across the economy: all of a sudden, the sidewalks seem cleaner, and trains run on time more often, traffic seems less congested, and latency in your favorite software product starts going down (with AI being turned loose on that legacy's software's code base that it gradually refactoring and optimizing in the background).
Effectively, what typically is happening due to entropy (decay, latency, quality, dirtiness) will start to move in the opposite direction due to automation and background AI.
This reversal of perceived entropy will start gradual, and then accelerate, and then on a day-to-day basis many things you touch in your daily life will be improving and then... singularity.
They were right, on a case by case basis. It meant a brighter, more enlightened future for me personally.
I was born in nowhere Appalachia with access to nothing. A third tier failed retail store was exciting to have access to; a McDonalds was exciting to have access to. The Web appeared like god damn magic cast down from the heavens circa 1994-1995. I had almost zero interest in computers growing up in the 1980s and early 1990s. I immediately understood what could be done with the Internet: I didn't need to ask permission, I could just immediately start creating anything I was capable of building on this new canvas; no licensing, no heavy government restrictions, no enormous bureaucratic walls to climb over: dial up and go. Go learn, go create, go chat with people from Australia or France, go meet thousands of people on WebChatBroadCasting or IRC, go become part of the Doom & Quake mod scene and learn how to program/script/design levels/whatever.
I think only highly privileged people would fail to grasp how extraordinary it was, they already had access to really nice things.
Why would that be a bad thing? Cases where corporation win, customers lose, are newsworthy because they're so much rarer than the mundane and taken for granted state of affairs where corporations make money by helping customers. Are your best examples of times that technological progress hurt us TV and the internet? They clearly helped a great deal, on net.
Which of course begs the question, "Where's all that entropy going, shouldn't it still be going up?"
That seems like the wrong approach? The claims of an article should be informed by all relevant sources, not the selection of sources be informed by the claims of an article.
This seems like an optimistic interpretation.
https://www.researchgate.net/publication/340531396_Automated...
If input a claim that is unsupported, and you search for support, isn't that a fundamental error? That is, is the software assuming the claim is accurate?
When SIDE’s results were shown to a group of Wikipedia users, 21% preferred the citations found by the AI, 10% preferred the existing citations and 39% did not have a preference.
and 30% didn't respond?I think the evaluation is flawed these subjective numbers would mean nothing if I did the survey. Looking at some Wikipedia pages in the SIDE demo (linked in comments here), it is clear that they in some cases fail to identify what claims are made in the article, and that the subjective choice of references are too binary.
I double check references on wikipedia in subjects where I have basic understanding, it is usually easy to find better references but it takes so much time. So very impressive.
Compare https://en.wiktionary.org/wiki/titty_bar .
What annoys me is that all the managers who were accruing technical debt were right. AI will clean it up for them, decades later.