Rare Persian manuscripts dating back to 13th century now online

The reason for the revolutionary change in Persian language was the Muslim conquest of Persia in 633 CE and the introduction of Arabic language and culture to Iran. Anything before the conquest was in a language unlike the modern Persian. For two centuries following the conquest, there is little written Persian available, so we don't know how exactly the transformation occurred. What emerged after the two centuries of silence was a language based on Old Persian but with many Arabic loan words and written in a modified Arabic alphabet. This language is comprehensible by modern readers.

It should be noted that comprehensible does not mean easy. Reading old poems for modern Persian speakers is as difficult as reading Shakespeare is for modern anglophones, but it is a far cry from reading Beowulf.

Old Persian is quite hard to read and understand for a modern Persian speaker, but if one spends a few weeks learning the writing system of middle Persian (called Pahlavi), they will be able to understand a lot of texts written before 500 AD.

https://en.wikipedia.org/wiki/Pahlavi_scripts

Similarly, English saw its fastest changing in the 12th century during the Norman conquest, when French became the dominant language of the ruling class.

In both cases, most of the core vocabulary and grammar survived, but with enough influence and new words that it is hard for speakers after the change to understand writings before the change.

My understanding from Persian friends is that Rumi is about as understandable to modern Persian speakers as his contemporary, Chaucer, is to modern English speakers.

Whan that Aprille with his shours soothe

The droughte of March hath perced to the roote

And bathed evry vein in switch liquour

Of which vertu engendyred is the flour...

Literacy, sacred texts and political stability are factors in language stability. So is the "one damned thing after another" path of history.

But to your question... I'd say English is the "outlier" or at least the example that needs explaining.

Old English isn't really English. A better name would be Saxon. It's far closer to other languages from the north sea mainland, and not at all similar to modern English. Frisian is still spoken, for example. It's basically a sister dialect to old English. Saxon was spoken by the "Anglo Saxons" who migrated to/colonized to Britain in the early mediaeval period. They barely wrote (Alfred's chronicles being a notable exception), and literacy was mostly Latin anyway. They were also a minority language for a long time.

Then the "English" (Anglo Saxons) lost political power to the normans. French came with the Norman Empire and Latin was the written & liturgical language. Various dialects of "Middle English" became "regional" languages alongside Irish/Scottish, Welsh/Cornish, some Danish, and possibly others. A lot of French got mixed in.

Eventually the Norman empire subsided or evolved into the British one, settled in England and "went native." They became the English nobility, and their posh dialect became early modern English (like Shakespeare).

Underlying a lot of this instability is the fact that Britain, Saxony and NW Europe generally was barbarian country. They (we) didn't live in cities, were borderline illiterate. Etc.

Old native traditions (ie, not Christianity) were remembered in epic songs and sagas, not books.

Languages evolve and get pushed up and down the pecking order much faster in barbarian country. Barbarian culture usually evolves much faster and more flexibly than conservative civic culture.

The parts of Europe colonized by Rome (the British outpost doesn't count) speak Latin today. None of the (more numerous) Roman conquests in "the east" speak Latin. The kept speaking Greek, Egyptian, Aramaic and such... languages that had been spoken and written in capital cities for thousands of years... even back then.

Also see: http://www.sufism.ir/MysticalBooks.php

English translations are here: http://www.sufism.ir/MysticalBooks(11)-en.php

"Rebel Sultans" by Manu Pillai is a great complement, focusing on the Deccan, with a lot of material on how Iranian culture and politics influenced Indian history.