Why didn't the author try the most obvious test of reading difficulty of all: Flesch-Kincaid? [1]
[1] http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readabil...
First, it's she, not he.
Second, because I hadn't heard about that obvious test at all back then. I never pretended to do a superserious scientific analysis but rather answer to the questions that came to my mind, by using a computer to validate hypothesis.
But many thanks for the pointers & suggestions, though! I was thinking about rebuilding this to make it realtime+interactive so that more than one text could be analysed/visualised, so I'll make sure to introduce the new tests.
I think I may have misunderstood the purpose of your article: I first read it as a serious attempt to use textual analysis to do a comparison of the comprehensibility of Tolkein's best-known works, in which case you fell way short of the mark by going no further than word counting. I think it started off like that, but on re-reading it I see that you say you hit a brick wall (paraphrasing) and decided to have some fun visualising your results so far; something you did very nicely. So perhaps I got the wrong end of the stick.
I thought about what you've taken on here. Doing things at the word level is pretty easy. Taking into account grammatical structure to get things like sentence lengths and clauses per sentence, or breaking words up into syllables (needed for F-K, for example) is considerably harder. I'm interested to hear what you come up with. Flesch & Flesch-Kincaid are US inventions so perhaps not obvious to everyone.
On he/she: I agonised over that for a good ten minutes over my breakfast. I read around your blog and your twitter page this morning for clues as to the appropriate pronoun. In the end I couldn't tell, so I went with 'he' because I stereotyped you. I nearly changed my wording to use constructions like "the author," "they," and various other mealy-mouthed alternatives but they were too ugly. So I did try, but I got it wrong. I apologise.
Flesch Reading Ease score, which is what I actually meant, does the same but with different coefficients to come up with a more granular difficulty score, usually in the range of 30-100.
They're both pretty arbitrary. The more I read up on this subject the more respect I have for the author's own attempts at an originality score. It's all subjective ultimately.
While I consider defaulting to 'he' to be legitimate and acceptable, I actually prefer the zie/zir gender-neutral pronouns when I think about it. http://santiago.mapache.org/nonfiction/essays/zie.html If I ever begin to agonize about the gender of the person I'm talking about, that's enough to kick me over into using GNPs.
The idea makes sense, but I'm unclear on how to actually measure it in a way that's normalized by page count.