N-gram Analysis of the New York Times Weddings Section

That said, it might be nice to have an optional plugin to easily discover annotations and add your own to pages as you browse.

One simple one that comes to mind here is that you need to analyze to what extent changes over the period of the data set are caused by underlying societal changes, versus changes in the NYT itself; the end result will be a mixture of those two changes, some of which may be magnifying and others offsetting. The 1980 NYT and the 2013 NYT are not the same newspaper, not edited by the same people, not sold to the same readership demographics, and not soliciting the same advertisers, so it's somewhat questionable to treat it as a stable proxy for a social group.

Another common pitfall is language change screwing up all kinds of measures (since n-gram models just work on word counts). For example, if two words are used roughly interchangeably in 1980, but by 1990 one of them has fallen out of usage, and been replaced wholly by the other one, searches for just the one word will look like the word's on an upwards trend, but it would be misleading to infer an increase in the underlying concept over the period. Of course, you can account for this by merging words into equivalence classes (most analyses will do basic stemming and merging of alternate spellings), but you have to be very careful to get all the equivalence classes (which is not a well-defined notion). Just a list of the top words in a year will tend to be some mixture of 1) top concepts; and 2) concepts expressed using only a small number of wording variations, so their count doesn't get diluted.

That's actually a slightly dubious analysis. The question you need to ask is 0.02% of what? In this case, I would take a guess it means 0.02% of all the words analyzed. As a very simple example, imagine analyzing all the letter in a book. If English were perfectly balanced, we expect to see all 26 letters at 1/26 or 0.038%, so seeing the letter 'e' appear at say, 1.0% (or even 0.1%) would be a notable statistical result.

I understand that, like college admissions, you can hire a wedding planner or consultant who can considerably raise the chances of your wedding being listed.

The NYT obits are another interesting read.

A day or two before the wedding, he told us that he wrote both a long form piece and a shorter, more typical piece. He wasn't sure which would get published, but he was obviously pushing for the longer piece to get in. It did.

Oddly enough, our write-up isn't included in the Rap Genius dataset. Maybe it's too recent or the longer write-ups aren't included.

http://ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2...