Inside the OED: can the world’s biggest dictionary survive the internet?
theguardian.com
theguardian.com
I announced the project at https://www.mail-archive.com/kragen-tol@canonical.org/msg001..., bought a copy of the dictionary, and spent a bunch of nights at the archive scanning its volumes. https://old.datahub.io/dataset/oed talks a bit about the available data halfway through the project. I did eventually scan the whole thing, and since then other people have contributed other scans. So far nobody has OCRed it and imported it wholesale into Wiktionary.
At some point I hacked together a web service that would show you the page image for a given word (perhaps after a few tries), but I don't have it running now. https://archive.org/details/oed11_201407 seems to be a downloadable program that does something similar.
It's really unfortunate that the past century of work at the Oxford University Press will be lost and have to be redone. Such is the price of copyright.
Adobe (and I’m sure many others) now preserve the image but place text behind it in a hidden layer or else use a pseudo ML process to create a font from the scan and fit it to the text with high accuracy (and low binary sizes).
Might be worth revisiting.
Edit: just saw the full mailing list url (on phone, hard to see anything) and realized who I am replying to. Hi! I have missed your writings! I recommend “only a constant factor worse than optimal” all the time to people.
[…]
A few days ago, I emailed to see if ‘mansplain’ had finally reached the OED. It had, but there was a snag – further research had pushed the word back a crucial six months, from February 2009 to August 2008. Then, no sooner had Paton’s entry gone live in January than someone emailed to point out that even this was inaccurate: they had spotted ‘mansplain’ on a May 2008 blog post, just a month after the writer Rebecca Solnit had published her influential essay Men Explain Things to Me. The updated definition, Proffitt assured me, will be available as soon as possible.”
One Wiktionary contributor did a better job[1] in 2012 by immediately finding the use[2] from May 2008. The OED is more Prestigious and Respectable and Authoritative, but the Wiktionary is more comprehensive, informative, reliable, convenient, useful, also cheaper.
[1]: https://en.wiktionary.org/w/index.php?title=Citations:manspl...
[2]: https://web.archive.org/web/20130518221612/http://www.journa...
If we define "better" by speed, then every HN comment is better than every book ever published, and all the blog posts on climate change are better than the scientific research. I find it's the opposite: The things that take longer to publish are usually better.
It is ironic that the definitive book on the English language was compiled by an American, and that he made it while he was incarcerated in an asylum for murder just makes the tale more interesting.
Simon Winchester is an excellent storyteller and the Surgeon of Crowthorne is an entertaining and insightful book - highly recommended!
Sadly, the archive only contains 300,000 words IIRC.
It's just "OED_unpack_tools.py" on github.
Unfortunately it's one of those things where the free version is "good enough", so most people won't pay for the improved version. Especially at $295/yr.
The Encyclopædia Britannica is the only thing I've (slightly) leafed through before I started really getting online. I remember its thin pages and dry reading compared to less formal encyclopedias. I even remember pining over advertisements of the CD/DVD set and thinking about whether it'd be better to connect to the internet at home, or get the DVD of Britannica.
https://www.amazon.co.uk/Compact-Oxford-English-Dictionary/d...
Yours in mildly wrecked condition for only 23 quid:
https://www.amazon.co.uk/Compact-Dictionary-COMPLETE-SLIP-CA...
For example, State Library of New South Wales.
The OED uniquely solves this problem through an unbelievable amount of scholarship into each word's range of meanings, providing incredible breadth and depth, back to the known beginnings of its usage. If you are serious about knowing what something specifically means, it's essential. As one simple example, I find it to be the best source for really grokking mathematical terms by far, and also sometimes the meaning of the 19th century mathematician is different than the usage in 2018.
1. Understand (something) intuitively or by empathy.
Origin: 1960s: a word invented by Robert Heinlein (1907–88), American author.
I knew the word, but I'm pleased to discover the origin.
By the time a term enters the OED, it's already fairly dry, or the editors are jumping on a bandwagon in an attempt to meme a word into existence for political reasons. Witness the Oxford English Dictoary's 'Word of the Year 2017': youthquake. https://en.oxforddictionaries.com/word-of-the-year/word-of-t...
Good luck trying to hold onto relevance with stunts like that.
I managed to not learn the definition of that "word of the year". Through all the news articles about their decision, the outrage, I remain in blissful ignorance, content with my superiority over the Steve Buscemis at OED.
"How do you do, fellow kids?"
Personally I enjoy and visit UD more often than OED.
The OED that's being referred to here is a 20+ volume set that costs about a thousand dollars: https://global.oup.com/academic/product/the-oxford-english-d.... University libraries will probably have a copy or two, and I suppose some high school (9th-12th grade) libraries might have a single rarely consulted set, but I'm doubtful any elementary schools in the US have one.
The best evidence I can give for its exclusiveness might be the link on the front page of the official OED site http://www.oed.com to the information about the print edition: http://public.oed.com/about/the-oed-in-print-and-on-cd-rom/. It goes to a page not found. Rather than being a standard companion book, apparently so few copies of this are sold that no one has bothered to fix the broken link!
http://www.oed.com/staticfiles/oedtoday2/oxford_and_the_dict...
The mobile app for "Shorter OED" has 600K words and costs $30: https://www.mobisystems.com/shorter-oxford-dictionary/
Sure on rare occasions you'd look up something that was left out, but that already happens in the other direction (towards rare words) because that and the are taking up space where more difficult words could have been included. Smaller dictionaries often don't have the word I wanted, but do have a ton of words I'd never need to look up.
[1] https://english.stackexchange.com/questions/42480/words-with...
And I don't think a rare use of a common word is any less likely to be looked up than a rare word.
https://www.amazon.com/Thinkers-Thesaurus-Sophisticated-Alte...
And its one of the reasons I'd like to know about the toolset used to construct dictionaries - I'd give anything to be able to throw a word set at these tools, have a custom dictionary constructed from it, and repeat until unity...
[1] https://www.amazon.com/Little-Oxford-English-Dictionary/dp/B...
I understand that unity is a difficult thing to attain in dictionaries, but perhaps there are tools and methods used by lexicographers that I need to learn about, as an interested party.
What else do you need? A hypergraph of word-vertexes and relation-edges animated in webgl, layered by categories and streamed from an elastic back end? That's your brain.
In printed form, or is web-based okay?
I don't. I do have a background in dictionaries, though: I created The Online Slang Dictionary (http://onlineslangdictionary.com/) which is the eldest slang dictionary and thesaurus on the web. I've given some minutes of thought to turning the underlying code into a product...
What sort of data would you like to capture in the DB, and what would apps do with it?
The reason is that I'm fascinated with dictionaries - the physical kind - in general, and would like to incorporate some sort of dictionary/glossary system in most of my apps as a means of documenting the technical terms for the subject those apps address.
Another thing is that I'm a complete newbie to the subject of lexicography, and I'd love to learn more from like-minded individuals, on what kind of tooling and methodology is out there. I suppose I could go and look at the sources for such things as the OpenDict, sdcv (https://wiki.archlinux.org/index.php/Sdcv) and so on .. but surely there must be standards for the ways Websters and so on construct their lexicography. I'm just not familiar with these tools, so I'd hoped to have some insight from other like-minded individuals in this community, before I launch into a google-trail of my own investigation.
With your site (wonderful, by the way) - did you invent your own schema for the database, or are you using some standard tooling on the backend?
My ideal tooling would give me the ability to pipe a text file into it, construct a list of all un-defined terms/words/symbols, give the means of importing definitions from known sources for each word, and so on. Ultimately I'd love to be able to construct custom dictionaries for any given corpus, such that no term in that corpus is undefined; and to be even more meta about it, no term in the dictionary itself would be undefined. Obviously this is a highly iterative thing, as the undefined terms list grows with every definition added - which is why I think I have to understand the tooling better before I start constructing my own processes for this task.
That is I don't even click at the links: the condensated summaries usually contain the definitions (from multiple dictionaries).
I'm not sure how this works out for the actual content providers.
and the double-tongued dictionary is fun: https://www.waywordradio.org/dictionary-listing/
What is disappointing to me is that, with so many more options, people have not adapted by utilizing only the best information. Anyone can read the best sources (with a major exception; see below), but instead they choose more and more crap and now even actively delegitimize the better sources in favor of propaganda. In other words, why hasn't the vastly increased competition in the marketplace of ideas yielded far superior knowledge? Epistemology should be one of the hottest words and hottest subjects of the day.
I hypothesize that it's a failure of the intellectual elite. Instead of spreading their incredible wealth to the world over this new medium, they kept it to themselves behind high paywalls (science journals, OED, JSTOR, etc.). And instead of defending the values of and passion for knowledge and intellect, of the Enlightenment, scholarship, and reason, many I see and talk to adopt the trendy anti-intellectualism, bizarrely undermining their own reason for being.
I know many people will say that they can get a dictionary for free, why use the OED. The problem is that those who know better are not standing up to assert why it is superior, necessary, and incredibly valuable to the world.
There are a couple of problems with this.
First, the OED is primarily a work of historical scholarship. Anyone in the market for a dictionary so that they can look up the meanings of words would be better off getting something else, like Merriam-Webster or really anything. The OED is not a good choice for this purpose.
Second, a story: I spent a semester studying Chinese in a foreign student program at a Chinese school. Like everybody else, I used Pleco for my dictionary needs.
My friends in the class quickly noticed that I was getting much better dictionary value from my installation of Pleco than they were getting from theirs. When one of them asked how I was doing it, I responded "I bought for-pay dictionaries" and they immediately lost interest.
There was no need for me to assert that the dictionaries I was using were superior and valuable -- my friends had already come to that conclusion themselves by watching me. They just weren't willing to pay $20 for that additional value.
I couldn't disagree more strongly. I use it to learn the meanings of words and there is no better source:
What I really love about the OED is how it keeps getting better even in its coverage of old words. There are lots of words which ten or twenty years ago had the etymology given us "unknown" but which now have a detailed explanation, thanks to recent research. (If you want an example, I think the word "koala" is one.)
That's simply not true. The OED has the current meanings, in more depth than anyone else.
> That question is relevant to almost no one.
Also not true. Most books and other texts were written before 2018.
The printed OED does not, though there have been separate print supplements to the print edition that get it closer.
The online edition, OTOH, does.
Tangentially, is there a better use case for moving to a database than 20+ volumes of books? To put it another way, if someone had a database and said, 'hey, let's print this thing out - it will be around 20 large books - and sell it!' ...
It is just chaotic is all I could infer from a first look. I mean English can be pretty messy already and maybe a specific Chinese dialect will be more regular than the bigger picture of the whole language. It's not that your coeds were being cheap, perhaps, it could just be disappointment for something as basic to cost anything at all, and relieve that it's not their personal shortcoming, but just an externalized advantage. You seem to say not even $20 was low enough, have I got that right at least?
Standard written Chinese is based on the grammar of spoken Mandarin. If someone mentions a Chinese dictionary without mentioning a particular dialect, they're almost certainly referring to Standard Written Chinese (essentially Mandarin).
Also, Chinese is less a single language than the Romance languages are a single language. The more far-flung dialects share less in common than, say Italian and Spanish. If Portugal were a province of Spain, Portuguese would probably be considered a dialect of Spanish. As my Linguistics professor used to say, "A language is a dialect with an army." One rarely deals with "the whole Chinese language", if by that you mean the union of all of the dialects. That would be like dealing simultaneously with Romanian, Portuguese, Romansh, French, Italian and Spanish as a single entity.
Dictionaries are not basic in the slightest. I'm constantly frustrated that the state of the art in Chinese/English dictionaries is still not very good.
Here's an example. The Mandarin word for "protect" is 保护 bǎohù. As a matter of semantics, protecting involves three roles: (1) the protector; (2) the beneficiary; (3) a danger to be warded off. In English, (1) is marked by being the subject of the verb protect, (2) is marked by being the object, and (3) is optionally marked by being the object of a complementary prepositional phrase headed by from: in
1E. I will protect her from going hungry.
A quality dictionary will include all of that information if you look up protect. But the state of the art in Chinese/English dictionaries is to note that 保护 is a verb, that it means "protect", and to provide a few example sentences, none of which feature role (3) at all. I had to ask a Chinese person how to indicate the danger involved in protecting, which it turns out is marked by an entire complementary clause:
1Z: 我保护她免挨饿 wǒ bǎohù tā miǎn ái'è
The OED is excellent for this purpose, due to its completeness, and other, even “unabridged” dictionaries fall far short of it in this use.
Of course, you have to a pretty particular needs in words to look up to be frequently hitting the space where the OED is needed for this.
But unlike most everyone here, I'm acknowledging that the value I get from the OED is purely entertainment value, not utility.
I'll note also that it obviously cannot be true that "many, many people" find the OED useful, as the total number of people who access the OED more than a couple of times a year is less than "many, many".
The premise of this comment is that everyone must agree with it (they just aren't admitting it). I'm sure there's a word for that ...
I don't access an optometrist more than once a year.
I find optometry very, very useful to me. Frequency of access and value derived from a thing have no necessary relationship.