Hacker News Reader

Comment by weinzierl | Hacker News Reader

The beauty of Encarta and its printed predecessors is that they are unalterable snapshots of the knowledge and views of their time.

I inherited an encyclopedia from the 1930s and it is a constant source of surprise and wonder to me - both in what is different from today and in what I expect to be different but is not.

I think Encarta is the last of this generation.

Sure, I can download a snapshot from Wikipedia and I did and others did and do as well. It's still not the same, because all our snapshots are different. Anything surprising in there could just be fluke, an editing error, the temporary state of an edit war.

A Wikipedia snapshot proves nothing. My 30s encyclopedia on the other hand is a stable reference. If I doubted anything in my copy I could easily find another antiquated copy and compare. Same with Encarta, there are so many copies out there that this particular snapshot of human knowledge will never die.

View on HN

Funny, everything you describe as a positive I see as a negative.

The 11th, 12th, and 13th editions of the Encyclopedia Britannica were from 1911, 1922, and 1926.

So if, as a researcher, you were interested in the state of knowledge in 1916, you're out of luck (if you're limiting yourself to a particular encyclopedia, for example).

I don't know what value you see in "stability", when the world's knowledge changes on a second-to-second basis. It's equally arbitrary whether you freeze knowledge at 1911 or in any given point in time since Wikipedia began.

When you say "A Wikipedia snapshot proves nothing", I have no idea what that means. If you're worried about vandalism or quick edits, it's trivial enough to compare with earlier and later versions of the page to ensure you're looking at relatively stable text. Heck, most of that could even be automated if you wanted.

And while Wikipedia articles are full of errors, so too were the articles in every edition of Britannica. The difference is that Wikipedia errors tend to get fixed a lot quicker, while the Britannica errors just remained on the paper they were printed on.

I genuinely don't see what difference there is between the "stability" of the Britannica 1911 edition, and the "stability" of Wikipedia at some arbitrary timestamp. Both capture a similarly arbitrary moment in time -- Wikipedia just gives you so many more to choose from.

> Heck, most of that could even be automated if you wanted.

I've had the idea of building something like this. The concept being that you would select an article and a time interval, and be shown the "best/most stable" revision of the article within the given window. The tool could use any number of metrics for determining which revision is best, the most reasonable one I've managed to come up with is "highest number of views during a state where the article was not locked/available to edit".

Yeah, the idea had never occurred to me until now, I'm glad to hear someone else has already thought of this.

I wasn't thinking as much as identifying any single best revision, but utilizing more of a diff-like tool to identify the text/changes that remained most stable over time -- where a brand-new edit doesn't count for much, but the longer it stays around as other edits are made, the more trustworthy it presumably is.

I think the biggest problem comes as articles get rearranged and expanded -- a section gets split into two or three, something gets moved from one section to a more appropriate one, and so forth. Or heck, sometimes entire articles get split into multiple ones, or vice-versa. I'm not aware of any diff-like tool/algorithm that handles these situations well, to accurately track how the same information gets moved when it's not just a simple case of insertion.

Huh, so a bit like git blame? And then you would merge together the chunks/edits which are most stable? That sounds awesome!!

I suppose you can't really count on the same text/markup being shifted around as articles get split and modified in the ways you've described. Also I suppose there is no such thing as a cross-article edit in MediaWiki terms iirc. Use vector embeddings? Throw an LLM at the problem? Rate editors on their familiarity with a given topic area (and track how that evolves over time)?

The idea of using edit information in addition to the raw text written by editors seems like it's extracting additional bits of information from human interactions.

I might have read some idea in a HN comment of training AI not just on code, but on how that code is edited in a git repo, or maybe I am just imagining it.

You should especially consider the role of a historian or the person who wishes to set out to correct facts and figures that are in error in Wikipedia, this is precisely what they need to correct it, to be able to trace the lineage of the error and correct the associated facts and figures like an excel sheet would, it’s very important especially with LLM’s that can’t cite their sources and google having gone to shit to find them, errors are quickly propagated across the web like some kind of crypto mixer, it becomes difficult to verify

A non-historian mathematician does a good job explaining the issue of census data errors, https://m.youtube.com/watch?si=ySApTldsYVf3jv0W&t=1630&v=GVh...

This is a bit off-topic, but how do you actually browse a Wikipedia snapshot? I tried turning a dump [1] into a browseable website hosted locally a few years back, and it seemed anything but trivial. Has any progress been made here?

[1] https://dumps.wikimedia.org/

Perhaps with something like Kiwix? I have used it for offline Wikipedia during prolonged travel...

https://wiki.kiwix.org/wiki/Main_Page

Seconded. It's a couple of years and there could be something better now, but I also used Kiwix. It's not perfect, but it works.

Wikipedia lists a bunch of offline readers (that I only found with Google)... https://en.wikipedia.org/wiki/Wikipedia%3ADatabase_download?...

Could you speak to the practical values of a stabilized reference encyclopedia?

If the study of history is a long-term, iterative process, where is this stability important (beyond the filtering of short-term noise, vandalism, political influence, etc)?

I can see several advantages: -having a "frozen" dataset, meaning that I can work on the same dataset as my colleague -Having all the topics frozen at the same date, representing global knowledge at one point. They knew X and Y, but Z was not advanced yet, so they could not do W. It is hard to realized the concomitance of knowledge bumps on the time scale in a continuously changing encyclopedia.

I think this is mistaking the costs of publishing with the stability of knowledge.

The knowledge was never stable, even back then. The encyclopedias could only afford a team of some fixed size, to publish an edition every few years.

Wikipedia just scales that up to many editors doing real time edits. Arguably it's a more reflective representation of how organic knowledge transfer actually happens.

Crowdsourcing knowledge from essentially anonymous contributors in this day and age is a terrible direction to be heading in.

As far as scaling up is concerned, Wikipedia has "power contributors" like any UGC platform. One guy alone has 3 million edits.

https://www.cbsnews.com/news/meet-the-man-behind-a-third-of-...

I remember having this argument 20 years ago, with teachers who've never seriously used the Internet.

I don't think it's still a realistic argument to be having in 2023 :)

Using Wikipedia as a citation in college is a straight ticket to a failing grade.

It's a secondary or tertiary source that does a great job or collecting and summarizing primary lit (which can be cited if more rigor is preferred, at the cost of context). My college professors encouraged the use of both, maybe with the caveat of citing a particular Wikipedia revision rather than the article directly.

But seriously, these were the discussions we've had two decades ago. If a professor today had such an issue, they're very old fashioned, and I'd probably walk out of that class because who knows what else that professor is out of date on.

So many fields today are evolving so rapidly, I wouldn't trust any single expert on a topic. Better to have a living crowdsourced reference that collects many sources.

Citing any encyclopedia or other general reference text as a source is a warning sign, because it means that you haven't done any in-depth research, and/or you're trying to cite sources for information which is considered general knowledge in the field. Wikipedia isn't special in this regard; you're going to get marked down just as much if you cite the Encyclopedia Britannica or a dictionary.

Reply on news.ycombinator.com