English Letter Frequency Counts: Mayzner Revisited

Unfortunately Twitter no longer allow redistribution of tweet content (http://readwrite.com/2011/03/03/how_recent_changes_to_twitte...). Datasift and Gnip offer commercial access to the full firehose, but for a sample it wouldn't take long to build your own corpus from the official Twitter streaming or search API.

I would think tweets would be skewed, no? Slang, memes, shrtnd txt 2 avoid char lmts, http://urls/, amongst others?

@chime: I am working on something similar and it would be great to exchange ideas if you want to chat about it.

You might be into corpus linguistics...

The results seem consistent with previous data. There are many recent layouts (e.g. Colemak[1]) which attempt further optimization, but simulations show little actual difference in strain (~5%) between those. They are all better than qwerty in this regard, but optimizing any any more is a clear case of diminishing returns.

It is actually much more difficult to model finger strain that the english language (in terms of n-grams). Subjective assessments vary a lot, and the quest for the optimal layout is bathed in controversy. Beyond switching away from qwerty, the most significant gains will be made by hardware solutions, like using an ergonomic keyboard (such as the very promising ErgoDox [2]). Other optimizations may come in the form of chorded keyers and better predictive technology. In this last case, the Google data may prove useful.

[1] http://colemak.com/

[2] http://deskthority.net/workshop-f7/split-ergonomic-keyboard-...

But people don't write just English words. Norvig dismisses any word with any character other than a-z, but those words still get typed. As do numbers, punctuation and not-real-words like names.

Redesigning a keyboard should be based on what people need to type, rather than on how English words are structured.

Norvig's spell corrector is the nicest code I've ever read. But every analysis he makes here is just counting, there is nothing computationally interesting happening. I doubt the code is really that much to look at.

The real order is this one...

http://en.wikipedia.org/wiki/ETAION_SHRDLU

A better strategy for hangman:

http://www.datagenetics.com/blog/april12012/index.html

As you might have seen, all those words with twenty letters or more are very weird. That’s because there aren’t very many commonly used long English words.

Also, at least 100,000 times isn’t so frequent, given Google’s huge corpus.

Here is the relative frequency of “Forschungsgemeinschaft” over time: http://books.google.com/ngrams/graph?content=Forschungsgemei...

Here are results on Google Books when searching for “Forschungsgemeinschaft”: http://www.google.com/search?q=%22Forschungsgemeinschaft%22&...

Google doesn’t promise that their English corpus only contains English books, just that it contains mostly English books. Given the raw size of their corpus this algorithmic solution seems like a reasonable tradeoff to me.

The frequency of “Forschungsgemeinschaft” is also easy enough to explain: The “Deutsche Forschungsgemeinschaft” (German Research Foundation) provides lots of funding for research in Germany. It is consequently mentioned in lots of research papers, many of which are written in English.

A lot of foreign words are frequently met in English texts, especially in narrow domains like philosophy, biology and medicine, etc.

And seing that:

http://en.wikipedia.org/wiki/Deutsche_Forschungsgemeinschaft

is a research institute, one would expect tons of mentions for this word in _english_ scientific papers.

But that is an outlier, and given the vastness of the corpus those would have been sorted out.

That said, there would have been an easy way to filter non English books out of the way automatically: do a statistical analysis on each book (e.g on letter frequency) and reject the ones that stray too much from the norm -- or send them to a secondary filtering stage, e.g by word presence or a verification by a human. Done carefully that filtering would not harm the actual results at all (e.g by presuposing a specific letter frequency, because it would only reject extreme outliers that would indeed by non-english works).