Stop Using the Flesch-Kincaid Test
andreadallover.com
andreadallover.com
For many their PhD thesis is probably both the most laborious and the most scrutinized piece of work they will have to produce.
When I was in high-school, I was in a camp with some American Peace Corps students in my country. They were learning Romanian since they had been spending some time here and were curious. At one point, one of them asked 'how do you say "hug"?' to which we replied 'îmbrățișare' (uhm-bruh-tzee-shuh-reh, approximately). They were taken aback, and after a pause, jokingly quipped 'don't you have a shorter word? Like... Hug?'.
Both are (through intermediaries at least in the case of English) coming from the latin prefix "in-" and "bracchia" (arm).
Also compare to modern Italian "abbracciare".
Just as a side note, 'brace' here (brațe in Romanian, braccia in Italian) is simply the word for arms (as in hands, not as in swords), so the word in romance languages sounds essentially like 'enarm' would in English (without the weapon connotation).
A similar transition is happening with Japanese, where English, the dominant world language in the 20th century and second language for many Japanese, is having its vocabulary widely adopted into the Japanese language. For example, the more common word for "ball" in Japanese is now "bōru" (literally, "ball" pronounced with Japanese syllables), while "tama" refers to more traditional items.
French, interestingly, is consciously hostile to borrowing vocabulary from other languages, to the point of having the Académie française in France [0] (and, to a lesser extent, the Office québécois de la langue française in Quebec [1]) act as "the authority" on the language, prohibiting borrowed words and issuing directives on correct usage and new vocabulary.
[0] https://en.wikipedia.org/wiki/Acad%C3%A9mie_fran%C3%A7aise [1] https://en.wikipedia.org/wiki/Office_qu%C3%A9b%C3%A9cois_de_...
The given counter-arguments against their use in general ("but what about context!", "they're clearly too simple!") have been tread over for years. Readability formulas are surprisingly robust, although obviously weak to adversarial input.
Here's an almost random, older paper that discussed readability score differences between genres: http://csjarchive.cogsci.rpi.edu/Proceedings/2008/pdfs/p1978.... Their conclusion:
... the analyses confirmed that several frequently used approaches for measuring vocabulary difficulty tend to be structured such that resulting text difficulty estimates overstate the difficulty of informational texts while simultaneously understating the difficulty of literary texts. These results can be explained in terms of the higher proportion of “core” vocabulary words typically found in literary texts as opposed to informational texts.
I wonder what William Buckley would have thought of the claim that the language of conservatives is simpler than that of liberals?
> the language of conservatives is simpler than that of liberals?
Times have changed.
If we should stop using this test, what should we start using? In the author's comment on the study, they noted "There are ways to study linguistic complexity".
I'm aware, for example, of this python project which provides F-K scores along with 7 other readability metrics to consider: https://pypi.org/project/py-readability-metrics/
The Boomerang extension for Gmail applies a test, in the free version, to give you a readability grade level. I always find this scores me at 12+ and I try to do better (lower score), because I know that there's nobody reading my emails and I will have wasted my time, unless I make it short and to the point.
And now I know the name of the test. Cool.
At what point does it become irresponsible to put out studies with this kind of title?
All this does is create two warring factions - one side that is bolstered by the claims and feels they are superior and the other side that feels attacked and loses some faith in the instition of studies like this.
Furthermore these are rarely ever conclusive. They take a topic like understanding and interpreting linguistics (an already incredibly difficult field of study) and boil it down to a dinner table conversation quip.
But the damage is done. This click bait will be freely shared on social media and be consumed by unwitting participants.
Politics is hard to study, but it's important. If the results are valid, people will apply them, and that has a direct effect on people's lives. The data may be messy and the theory doubtful, but if there's any measure of merit to it, it can gain a small advantage that can be the critical difference. And hopefully, each subsequent study builds evidence for a paradigm shift that makes the theory less dubious over time.
To truly annul the study, the author needs a smoking gun - for example:
- Take a couple of the source texts that score wildly far apart
- Rework the punctuation such that an objective reader still agrees that the transcription is a valid representation of the original speech
- Show that the readability scores in the re-punctuated texts are inverted - or at the very least, score within a small percentage of each other.
It's a bit like trying to measure the happiness of a city by counting the percentage of smiling people as they walk down the high street. There are many social reasons a person might smile from the depths of misery, different people might classify a smile differently, and different cultures might view walking down the street with a grin as either frowned upon or mandated. Weather, time of day, month of year, public holidays - etc - all will affect the results.
But so long as you work hard to eliminate these biases, you still couldn't use such a measure to test whether Beijing is happier than Belfast, but you might use it as an indicator of whether left handed people are happier than right handed people. You can't tell whether men are happier than women (social pressures will influence facial indicators along gender lines from an early age in many cultures) but you might be able to begin a conversation on whether conservatives are happier than liberals. It won't be conclusive proof, but as a starting point for future research, it still holds some validity.
My point is that nothing in this criticism conclusively proves that this particular measure is unsuited for measuring against the stated outcomes. Even a measure that is the current darling of the linguistics community will have its drawbacks, and the author needs to prove that the weaknesses in the measure present a bias in and of themselves, not just that they are generally deficient.
To my mind, the better criticism is intent. A person can - and should - craft a speech based on the audience. A speech given at a university or research facility should have a different complexity score than one given at a rally or disaster zone. The interesting question to my mind is whether more effort is put into this by conservatives or liberals, and then comparing that to those in power vs their contenders.
But then, that paper wouldn't have such a catchy title, would it?