> Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state.
https://arxiv.org/abs/1806.08053
Roughly speaking: The amount of computation or energy needed to perfectly reproduce a random source, such as a coin flip, is high, while the significance or meaning, for the average receiver, is low. Natural language text requires less computation to reproduce [1], but, for the average receiver, the significance is higher.
Also, what about crystalline forms, which are very orderly and require minimal computation to reproduce, but are equally insignificant for the average receiver?
More or less correct. The key difference is that you could not compress a random coin flip sequence (and that a compressed text is meaningless until decompressed to original).
> all minimal programs are by definition Kolmogorov random
Compression provides an upper bound to K. Kolmogorov Randomness itself is not computable. AKA: You can't ever know if you have a minimal program.
> Crystalline forms
It is possible to both have low significance and low information content. Crystalline forms were very significant to Turing though: https://en.wikipedia.org/wiki/The_Chemical_Basis_of_Morphoge...
With this in mind, you can see text as the output of an algorithm (brain) taking many such decisions. The information entropy contained in this text reveals the complexity (amount of distinct binary decisions) which needed to take place in the machine (brain) in order for the text to occur.