Researchers: It takes 1.5 MB of data to store language information
medicalxpress.com
medicalxpress.com
It has been done on small models, though, at least to some significant extent: https://en.wikipedia.org/wiki/OpenWorm
A full fly connectome: https://www.nature.com/articles/s41684-018-0183-8
Architecture of the Mouse Brain Synaptome: https://www.cell.com/neuron/fulltext/S0896-6273(18)30581-6
So we are past just a simple worm simulation now.
This page shows some of the current theories: https://en.wikipedia.org/wiki/Neural_coding#Hypothesized_cod...
That said, they assume a computer. and a program to run it. Similarly, the brain probably has many complex lower structures that compress language storage, but the compression software is not trivial to store itself.
I think the storage requirements for the semantics is in particular questionable. If you can refer to a particular concept somewhere in the brain you only need a pointer to it. However, single pointers might not be sufficient
The apparatus to learn a language might also not so easily distinguished from the structure to store the language or the structure to quickly recall the language. In that case taking about storage is like talking about compressing Plato's world of concepts.
It describes how to summarize a language from a information theory standpoint (which is something different).
There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation (and even if not, it's a good lower bound)
Do you have a citation? I would be very interested to see it.
I hate to make analogies between computing and biological systems, but I immediately think of denormalization in databases, error correcting codes, and database replication as examples of optimizations that take you farther from information theoretic efficiency.
Information applies very straightforwardly to the study of a spike train of a synapse (source, channel, and receiver). I remain unconvinced that it systems like memory make sense as information channels.
https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
The summary at wikipedia:
https://en.wikipedia.org/wiki/Entropy_(information_theory)#E...
Notably, PPM, the best compression algorithm (at the time), was not able to predict as well. So that's a quantitative demonstration of optimality in language performance that includes a faculty for understanding syntax and semantics and presumably is linked to phonology, pragmatics etc.
Chomsky discusses optimality in a 1998 talk here:
https://youtu.be/7Sw15-vSY8E, at about 1h 7m an onwards.
He says language evolved rather quickly for such a significant function (estimating it arose quickly ~200kya). He says though many things in evolution are pretty messy, language is close to optimal design. He doesn't provide references, but I think (given the time and academic associations; anyone know?) he may be referring to Optimality Theory:
https://en.wikipedia.org/wiki/Optimality_Theory
Separately, our knowledge of anatomy of the language faculty is that it is highly localized in the Wernicke and Broca areas of the brain. This also suggests a small modification of an existing brain structure.
Additionally, optimality in biological systems has been a productive research direction in a few areas: - Optimality Principles in Biology - https://link.springer.com/book/10.1007/978-1-4899-6419-9 - Near optimal energy use in the metabolism of cell division - https://arxiv.org/pdf/1209.1179.pdf - https://www.quantamagazine.org/the-math-that-tells-cells-wha...
I poked through the optimality literature in biology pretty thoroughly a few years back. There is a great deal of interest, but except in a few cases where a simple piece of natural history is nicely described by an evolutionary game, the choice of criterion to optimize is generally arbitrary. Without an evolutionary mechanism and observations of forces driving that mechanism there is no reason beyond just-so stories to apply any particular criterion.
I think we're at a point with language and its neural mechanism where by analogy it looks like the phenomenon of flight being closely determined by the exact form and use of a wing. Almost any shape will not do, so there is a form closely (efficiently) following function. Of course there were preadaptations for both which provide very inefficient flight or communication abilities, but something flips and the adaptive utility of the function rapidly sculpts and elaborates the form in that direction.
About optimality, generally agree though I think following thermodynamic efficiency seems a deep inductive step as that allows measure of what is arbitrary or not, e.g. in the work of England that I cited.
That is a bold hypothesis, certainly, but not one that you could use as a basis for argument at this point in time. I also deeply doubt it when viewed in the context of Wittgenstein's language games. A human and a sheep dog can very effectively engage in a language game, even subtle ones such as naming. Nor is it pure behavioral conditioning, as what makes various dog breeds better at particular things is selection for incidence of particular behaviors.
Then you go to a gorilla or a parrot, which can engage in quite abstract language games. Language games with parrots always make you aware of Wittgenstein's lion, though.
For Chomsky's hypothesis to be true, you would need to have a particular inflection point where language games become language, and for that inflection point to be reified as the evolution of a specific neural structure instead of exaptation of lots of aspects in the brain.
It feels too much like Watson and Crick's nonsense "one gene = one protein," which could have been discarded with a modest amount of thought before it was allowed to damage biology for decades.
> I think following thermodynamic efficiency seems a deep inductive step as that allows measure of what is arbitrary or not
Until you look at, say, bower birds and realize that conspicuous waste is a signalling mechanism for fitness. There you have an evolutionary game that rewards thermodynamic inefficiency.
People used to think that about genomes, until we got to know in some detail what's in them.
The non-coding genome is a large part of what causes us to look different from a mouse or a fruit-fly. Gene regulation (what gene to transcribe, and when) is largely controlled by non-coding DNA: https://en.wikipedia.org/wiki/Regulation_of_gene_expression
There are energy costs associated with useless information being stored and transcribed in the genome... so it is pretty reasonable to believe that information theory does play some (however minor) role in how data is stored.
The fact that the DNA involved in gene regulation is mostly non-coding does not mean that most non-coding DNA has a regulatory purpose. If it did, changes in non-coding DNA would be more important, in aggregate, than changes in coding DNA are, with respect to evolution and disease, on account of its prevalence.
Most non-coding DNA does have a regulatory purpose. Things like copy number variations (where information is "duplicated") between promoters and enhancers can can change the rate at which particular genes get transcribed. This is my point: that viewing the genome as "information" in the simple sense of each ATCG nucleotide corresponding to bits is absurd. The genome is a geometric AND chemical entity and encodes information using both paradigms.
My point is not to say that you are wrong, but rather that you do not know. It is FAR from well established that the genome is inefficient at storing information.
From an information-theoretic point of view (which is where we started) even if all non-coding DNA had some purpose, it would not be a sufficient basis for assuming that it is close to the most information-theoretically efficient implementation. On the other hand, the fact that whole genomes can be handily compressed, with straightforward Huffman encoding techniques, settles the infomation-theoretic efficiency question, regardless of whether your speculation about the usefulness of all non-coding DNA is correct.
> The non-coding genome is a large part of what causes us to look different from a mouse or a fruit-fly.
Now that's just fantasy. There have been a lot of changes since our common ancestors with mice and fruit flies. We are all animals and obviously would share genes that perform analogous functions -- like common genes for controlling wing shapes and organ shapes. But would you like to point me to evidence in the human DNA that codes for fruit fly wings? And what role do you think having virus DNA in our genome plays in our development?
DNA encodes information. Information theory does play a role. But as mannykannot pointed out, the topic of the conversation is something else.
Paper: https://royalsocietypublishing.org/doi/full/10.1098/rsos.181...
OSF Page: https://osf.io/ga9th/
I've been actively learning Portuguese and Russian lately, it's impressive how much faster I can pick up Portuguese vocabulary vs. Russian. And that's even for words that don't have an obvious cognate in languages I already know. The structure of the words, the various building blocks are just so much more familiar in Portuguese. A word like "atrever" (to dare) doesn't have any obvious cognate in languages that I know but it just "looks right" it a way that, say, "atverer" or "aterver" wouldn't. Those last two words sound distinctly un-Portuguese (I might say, un-Romance) to me. That makes it a lot easier to remember the spelling.
Eventually as I grow my Russian vocabulary I start making similar connections. Волноваться is pretty tricky to memorize on its own but it becomes easier when you know that ~ся is the reflective, ~ть is the common verb ending, волна means wave and ~овать is a very common building block for Russian verbs.
Steven Pinker's book Words and Rules is a great layman-oriented read if this sort of thing interests you.
So, they're like "constants" vs calling a function to calculate a value.
I think a decent real example of that is fiancé/fiancée, those are french borrowings and have, at least originally, kept the French grammatical gender inflection. However nowadays I often see people using either spelling in a gender-neutral way since most people don't bother to learn French grammar for this one word.
That still wouldn't explain the why of having it like "went" vs "goed".
Sure, it's easy to memorize because we use it all the time, but why have it to memorize it in the first place, versus something like "goed".
So, this theory (I tried to convey above) said that it being irregular placed ensured we don't slow down try to derive it from regular rules, but instead have fast access to a memorized form.
Couldn't we just memorize "goed"? If it's just "frequency of use" that mattered, "went" and "goed" would work just as well.
But the extra idea is that "goed", being regular, would be too easy for us to confuse with thousand of other regular verbs, and not use our "fast recall" mechanism, regardless of that verb being needed all the time.
Not sure if correct - read it years ago. This seems to be related to that:
https://en.wikipedia.org/wiki/Regular_and_irregular_verbs#Li...
Natural languages are not designed, they evolve. The irregular nature of the conjugation of "to go" might just be a remnant of some archaic form and nothing else, in the same way some argue that the plural of "octopus" is "octopi" or "octopodes. Does it serve any linguistic purpose? I don't see how, but it won't stop some people for saying it. Look at the use of the subjunctive in "if I were you", which is one of the only occurrences of the subjunctive outside of set-phrases in day-to-day modern English. Is it really necessary? If one was to say "if I was you" would it lose some additional nuance or meaning in practice?
I often see people trying to rationalize some language features (such as arbitrary genders of nouns in many languages) as error correction or some "optimization" but I'm generally unconvinced. Maybe "I went" is just the linguistic equivalent of a platypus, some weird byproduct of a very long evolution with no other intrinsic purpose in the grand scheme of things.
But that's part of my point. I don't say irregular verbs were designed to be effective that way, but that they were evolved to be effective.
Hence the link with "language acquisition" being involved -- when regular and irregular verbs developed that wasn't a known theory some "language designer" could consciously follow. Just something innate that could develop because of a evolutionary advantage.
In fact, if someone merely designed, they'd probably go for all regular, rather than regular + irregular, as it's "cleaner".
>I often see people trying to rationalize some language features (such as arbitrary genders of nouns in many languages) as error correction or some "optimization" but I'm generally unconvinced.
Tons of language features are indeed optimizations for different things. Cold climates for example have languages with less vowels (keeping your mouth closed more).
It wouldn't have anything distinctive for people to latch on to, so they would be constantly trying to derive it from the general rules for regular verbs.
E.g.
(a) all verbs regular -> instinctively go to (slower) rule derivation instead of memorization of all verbs, even for the most common ones.
(b) most frequent verbs being irregular -> instinctively retrieve them from the (faster) "lookup table" of memorizations, and bypass the rule based derivation for them.
I.e having the clear distinction of irregularity makes it faster to go directly to that kind of "constant" memory.
That said, this is not my theory, read it years ago in a cognitive/linguistic pop science article. This seems to say more or less the same thing:
https://en.wikipedia.org/wiki/Regular_and_irregular_verbs#Li...
In studies of first language acquisition (where the aim is to establish how the human brain processes its native language), one debate among 20th-century linguists revolved around whether small children learn all verb forms as separate pieces of vocabulary or whether they deduce forms by the application of rules. Since a child can hear a regular verb for the first time and immediately reuse it correctly in a different conjugated form which he or she has never heard, it is clear that the brain does work with rules; but irregular verbs must be processed differently.
~ов is an iterative suffix - like ~le or ~er in gamble and chatter. It's useful to know, because you can rationalise why it is always dropped in the present tense - you can't be iterative at the moment, unless you're an Englishman)
I didn't know that, but you'll grant me that it's not a very useful cognate (either in spelling or in meaning).
>~ов is an iterative suffix - like ~le or ~er in gamble and chatter. It's useful to know, because you can rationalise why it is always dropped in the present tense - you can't be iterative at the moment, unless you're an Englishman)
Very interesting, thanks.
Considering that, 40k words to 400kbits is not too surprising.
Converting a large word list to a small number of bits has been a computer science hobby for a long time. Here's a pretty good search result to start working through for more details: https://duckduckgo.com/?q=building+a+small+spell+checker+suf... It was especially important to write small and fast spell checkers in the 1980s and early 1990s, when you couldn't expect to have enough RAM sitting around to simply load up a naively-encoded list of words, and the act of spell checking a few thousand words could take noticeable time.
So in an encoding scheme chosen to represent English compactly, I'm not too surprised that you can get things down quite small.
However, the question is, what relevance does that encoding scheme have to the human brain? Having just scanned through the paper, the answer is "probably not that much", which the researchers are well aware of. They explicitly present this as a lower-bound, which is a reasonable thing to do. It is obvious that the brain does not simply store 1.5MB of data in the way a computer does, in many ways.
To be honest, this amounts to an exercise in recreational mathematics more than anything else. There's nothing wrong with that, and that's not a criticism. My point is that I'm not sure it's worth trying to read the paper as anything else.
Now a few questions: Can you hear a word you've never written (in a language that you're familiar with) and intuitively spell it right the first time? Can you read a word that you don't immediately understand and figure out its meaning from the context in which it is used? Can you accurately complete half of a sentence?
That a lot of people can do these things suggests to me that we all sit on something superficially similar to an efficient lossy compressor in our brain.
Preferably, the floppy disks should be the 5-1/4 inch 1.2 MiB high-density, double-sided type, 3-1/2 inch disks sounds good too ;-)
Well, this "feature" creates significant compatibility issues as all the vendors created their own proprietary formats, sometimes with patents. As a result, together with the economics of scale, often, the standard format is not the best design, a simpler design that serves as the lowest common denominator.
On the other hand, due to this feature, it was already possible to store 1,760 KB of data on a 3-1/2 inch HD floppy on a 1986 Amiga. And in 2000, it was possible to store 32 MiB of data on a standard 1.44 MB floppy disk, by using a SuperDisk LS-240 drive (although random write is sacrificed, the entire floppy must be rewritten if a change is needed, like CD-RWs).
I believe the advancements in magnetic recording technology in the past 20 years allows one to achieve even higher capacity on a standard floppy, and it can be as cheap as early floppy drives if mass produced, only that it doesn't make sense to do so.