Hacker News Reader

Wikipedia is fairly reliable if our standard isn't a platonic ideal of truth but real-world comparators. Reminds me of Kant's famous line. "From the crooked timber of humankind, nothing entirely straight can be made".

See the Wikipedia page on the subject :)

https://en.m.wikipedia.org/wiki/Reliability_of_Wikipedia

The sell of Wikipedia was never "we'll think so you don't have to", it was never going to disarm you of your skepticism and critical thought, and you can actually check the sources. LLMs are sold as "replace knowledge work(ers)", you cannot check their sources, and the only way you can check their work is by going to something like Wikipedia. They're just fundamentally different things.

> The sell of Wikipedia was never "we'll think so you don't have to", it was never going to disarm you of your skepticism and critical thought, and you can actually check the sources.

You can check them, but Wikipedia doesn't care what they say. When I checked a citation on the French Toast page, and noted that the source said the opposite of what Wikipedia did by annotating that citation with [failed verification], an editor showed up to remove that annotation and scold me that the only thing that mattered was whether the source existed, not what it might or might not say.

I feel like I hear a lot of criticism about Wikipedia editors, but isn't Wikipedia overall pretty good? I'm not gonna defend every editor action or whatever, but I think the product stands for itself.

Wikipedia is overall pretty good, but it sometimes contains erroneous information. LLMs are overall pretty good, but they sometimes contain erroneous information.

The weird part is when people get really concerned that someone might treat the former as a reliable source, but then turn around and argue that people should treat the latter as a reliable source.

I had a moment of pique where I was just gonna copy paste my reply to this rehash of your original point that is non-responsive to what I wrote, but I've found myself. Instead, I will link to the Wikipedia article for Equivocation [0] and ChatGPT's answer to "are wikipedia and LLMs alike?"

[0]: https://en.wikipedia.org/wiki/Equivocation

[1]: https://chatgpt.com/share/67e6adf3-3598-8003-8ccd-68564b7194...

Wikipedia occasionally has errors, which are usually minor. The LLMs I've tried occasionally get things right, but mostly emit limitless streams of plausible-sounding lies. Your comment paints them as much more similar than they are.

In my experience, it's really common for wikipedia to have errors, but it's true that they tend to be minor. And yes, LLMs mostly just produce crazy gibberish. They're clearly worse than wikipedia. But I don't think wikipedia is meeting a standard it should be proud of.

Yes, I agree. What kind of institution do you think could do better?

It's scored better than other encyclopedias it has been compared against, which is something.

Wikipedia is one of the better sources out there for topics that are not seen as political.

For politically loaded topics, though, Wikipedia has become increasingly biased towards one side over the past 10-15 years.

source: the other side (conveniently works in any direction)

> Whereas I often see people treat Wikipedia, posts on AskHistorians, YouTube videos, studies from advocacy groups, and other questionable sources as if they can be relied on.

One of these things is not like the others! Almost always, when I see somebody claiming Wikipedia is wrong about something, it's because they're some kind of crackpot. I find errors in Wikipedia several times a year; probably the majority of my contribution history to Wikipedia https://en.wikipedia.org/wiki/Special:Contributions/Kragen consists of me correcting errors in it. Occasionally my correction is incorrect, so someone corrects my correction. This happens several times a decade.

By contrast, I find many YouTube videos and studies from advocacy groups to be full of errors, and there is no mechanism for even the authors themselves to correct them, much less for someone else to do so. (I don't know enough about posts on AskHistorians to comment intelligently, but I assume that if there's a major factual error, the top-voted comments will tell you so—unlike YouTube or advocacy-group studies—but minor errors will generally remain uncorrected; and that generally only a single person's expertise is applied to getting the post right.)

But none of these are in the same league as LLM output, which in my experience usually contains more falsehoods than facts.

> Currently, there's a much greater understanding that LLM's are unreliable.

Wikipedia being world-editable and thus unreliable has been beaten into everyone's minds for decades.

LLMs just popped into existence a few years ago, backed by much hype and marketing about "intelligence". No, normal people you find on the street do not in fact understand that they are unreliable. Watch some less computer literate people interact with ChatGPT - it's terrifying. They trust every word!

Look at the comments here. No one is claiming that LLMs are reliable, while numerous people are claiming that Wikipedia is reliable.

Isn't that the issue with basically any medium?

If you read a non-fiction book on any topic, you can probably assume that half of the information in it is just extrapolated from the authors experience.

Even scientific articles are full of inaccurate statements, the only thing you can somewhat trust are the narrow questions answered by the data, which is usually a small effect that may or may not be reproducible...

No, different media are different—or, better said, different institutions are different, and different media can support different institutions.

Nonfiction books and scientific papers generally only have one person, or at best a dozen or so (with rare exceptions like CERN papers), giving attention to their correctness. Email messages and YouTube videos generally only have one. This limits the expertise that can be brought to bear on them. Books can be corrected in later printings, an advantage not enjoyed by the other three. Email messages and YouTube videos are usually displayed together with replies, but usually comments pointing out errors in YouTube videos get drowned in worthless me-too noise.

But popular Wikipedia articles are routinely corrected by hundreds or thousands of people, all of whom must come to a rough consensus on what is true before the paragraph stabilizes.

Consequently, although you can easily find errors in Wikipedia, they are much less common in these other media.

Yes, though by different degrees. I wouldn't take any claim I read on Wikipedia, got from an LLM, saw in a AskHistorians or Hacker News reply, etc., as fact, and I would never use any of those as a source to back up or prove something I was saying.

Newspaper articles? It really depends. I wouldn't take paraphrased quotes or "sources say" as fact.

But as you move to generally more reliable sources, you also have to be aware that they can mislead in different ways, such as constructing the information in a particular way to push a particular narrative, or leaving out inconvenient facts.