How much innate knowledge can the genome encode?
wiringthebrain.com
wiringthebrain.com
There's a new soon to be hot (I predict) topic in AI/ML called inductive bias. It turns out that the way your neural net is structured, i.e. the overall shape and the way the neurons are connected, can strongly bias which information the neural net tends to learn, and how quickly. To the point that there was a paper (can't find it now) where IIRC the authors were able to develop an algorithm which creates neural nets which can solve basic problems with little to no training. Point being, you don't need to encode, say, the image of a nipple and the action of suckling in DNA to create innate suckling knowledge - you simply need to structure the brain such that it is biased to perform and learn certain sequences of behaviors.
I think this may be the origin of much if not all innate knowledge and I believe that research in AI/ML will lead to better understanding of human (and animal) cognition.
And indeed, what people will find is exactly as you say: The DNA encodes how to grow the network such that it will respond with the evolutionary advantageous instinct.
I don't know if I'd quite say that no one is looking into that. Obviously we know that there are NN that use deep networks. But also we can see that we can increase the computation power by removing nodes and edges that don't contribute to the correct answer. In evolution, this is a literal cost savings in that you need to feed less neurons.
On the molecular level, proteins are only functional at certain temperatures, pressures, and pH levels.
On the cell level, certain proteins might only be found in specific areas, and essential amino acids might be harvested from the environment rather than produced by the cell itself.
On the organism level, we are predisposed to shiver if it's too cold or sweat if it's too hot, an uncomfortable feeling, so innately we go to where the climate is milder and won't kill us. We clothe ourselves and that range is expanded, suddenly there is a selective advantage for being able to clothe yourself, just like there is to be a white beetle on a white sand beach instead of a green one visible to predators, or a cell that's able to just mooch someone elses amino acids rather than expend energy making their own.
So, technically, sure, but doesn't the busy beaver sequence imply the potential of anything more complicated than a 5 state turing machine is literally unimaginable? I mean, unimaginably huge even as a multiple of the size of the universe.
If we want to measure the information limit of DNA, we can argue the DNA is the program input... sure, so far so good. But then what is the "tape"? The tape must be empty at the beginning of the program... So, I guess we really need to consider the "program" is the entire universe of atoms...
Ok... but then... who said that the program ever Halts, exactly? Busy beaver needs to Halt.
So, we actually broke two of the fundamental assumptions of the Busy Beaver problem. So I wouldn't make any guarantees about the complexity of DNA based on it.
How do you think "structuring the brain such that it performs suckling" differs from "encoding the action of suckling"?
You just said the same thing two different ways.
The structure I'm describing is at a different scale from the structures of the individual neurons. Actually, come to think of it, baby motion may be an even better example:
When a baby is born, it (possibly) has no understanding of how to move it's arms and legs. But the newborn brain is structured in such a way as to encourage movement of limbs, such that the movement required for learning becomes a high probability event°. In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment.
If you connect neurons in a particular large scale pattern, and wire some up to ears, some to eyes, some to touch receptors, two things may happen:
1. Your brain is more likely to perform the initial actions required to learn
2. Your brain may more easily learn certain actions
The second thing is inductive bias, and I think it will be extremely powerful in the world of ML once we figure out how to harness it. We know that neural nets are hugely wasteful now, there have been papers published on neural network optimizers which can shrink nets by some 80% while maintaining accuracy. Inductive bias is probably such an optimization.
That is all of the information necessary for grasping. By your own stipulation, it is sufficient to enable grasping. Nothing more can be necessary.
What you're talking about is the degree to which the information is compressed at rest. But index.html and index.html.gz don't contain different information; they contain the same information encoded differently.
The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. This is what one would expect given what we already know about biology and Turing machines. The difference is the Turing machine here doesn't start with an empty tape -- far from it.
Why not?
Consider birdsong. Many birds must learn "correct" song from models. If raised without good models, they still sing, but they sing a deformed song. This is, correctly, treated as evidence that the song of these birds is a cultural phenomenon -- a bird raised in isolation will sing in an identifiably different way than a bird raised normally.
So is normal song arbitrary? Can we say that it isn't encoded in the birds' genetics? Those answers are no and no; when birds with deformed song raise children, those children learn normal song. Normal song is the only steady state of the bird population; any other state will inevitably become normal song.
This is sufficient to say that normal song is latent in the birds' genome -- the genome will create it from nothing if necessary -- even though the genome cannot cause a single individual bird to sing correctly in isolation.
Consider human language. Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not.
(Strictly speaking, the genome can't even create the bird from nothing, but I understand what you are getting at.)
Which? I would say the answer is clearly yes. The closest analogue we have is the innovation of sign language by deaf children who were brought into schools with each other. They created working sign language from -- again -- nothing; they could not learn spoken language due to being deaf, and they had no sign language model to learn from. This took only a single generation, even better than the birds.
https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language
This is an exact match to your hypothetical question if you're willing to overlook the fact that sign languages are not "spoken".
Nicaraguan sign language was not developed in a vacuum, it was developed by children living in a completely normal human society with the full use of all their senses but one. Far from "nothing".
Why do you think differently? "I think the very idea is ridiculous" isn't much of an argument.
If they are living in a normal society, they are exposed to language nearly continuously, from visible printed language to visible body language to the visual evidence of language being spoken all around them.
But I'm left disappointed that there's no actual estimate of maximum information content. Just this:
> *[Note from Tony Zador: The length of the genome, which in humans is around 3 billion letters, represents an upper bound on the Kolmogorov complexity. Only a fraction of that carries functional information, however, so the upper KC value may be quite a bit lower than that].
That must be 3 x 10^9 base pairs, in the haploid genome. Wikipedia tells me that's 800MB.[0] However, only 1%-2% of that codes for proteins.[1] Which is basically what Zador's note says.
So what's possible for the rest? While 800MB doesn't seem that much, maybe it's enough for basic "firmware" and "software". Including instinctual behaviors.
But even all that is clearly not enough for a full blueprint and assembly instructions. That, as TFA explains, must flow from interactions during development.
https://upload.wikimedia.org/wikipedia/commons/4/4b/Chromati...
each of these have huge implications for what can be done with the DNA
there's tons more information that's attached to the strands of DNA, too, like methylation or promoters, etc. that the simple ATGC doesn't cover at all
if we want to stretch the turing tape metaphor, it's more like marking the symbols as various kinds of magnets on a string, then putting a bunch of them in a sack and shaking them up. does that change the amount of information?
if you're interested, you really can just pick up an intro bio textbook, skim it, then do an intro genetics text (and skip the classical genetics crap).
How many orders of magnitude?
I don't know how I would quantify that, but there's a lot going on.
Of course, if we stay with that analogy, DNA would not be a "series of instructions" but would more resemble a huge, unordered set of "if/then" rules or "trigger/action" pairs - the "actions" being protein-coding sequences and the "triggers" being their regulatory sequences.
The actual behavior of the cell is then determined by the interation of those rules and the effects of the coded proteins.
DNA shape certainly matters for gene expression. Just simply accessibility of sequences. But also, as TFA notes, through binding of RNAs and proteins.
But still, it's sequence that determines shape. So I'm not sure how that could increase coding capacity.
It means nothing, in terms of information content.
This is a common misunderstanding, which the author of the piece appears to share, but there's no getting "more" information out of something by talking about its interactions. Suppose we write a computer language where every byte is a meaningful opcode, and all possible strings of bytes are valid and do something. If we consider the set of 8-byte programs, there are 2^64 of them. It takes 64 bits to encode which one you are choosing from that set, and there are 2^64 options. The fact the they are "programs" and thus that many of them will "interact" does not mean that you suddenly have more than 64 bits of information. You still have just 2^64 options. None have appeared from the ether by the virtue of some of them "interacting". In that set of programs will be everything from programs that immediately terminate, to infinite loops, to programs that use staggering amounts of RAM and CPU before terminating, but there's just 2^64 of them.
Information is really just a count of possibilities. (Weighted, technically, but that's not a terribly helpful detail at the moment.)
Since the question is explicitly does DNA itself encode innate knowledge, it is relevant to discuss the contents of the DNA in this strict counting argument. (Add the epigenetics and whatever else you like; all it does is add more bits proportionally to the possibilities they add. But they don't make more bits than that due to "interactions".)
You, yourself, are not the sum total of information in your DNA. Even at the one-cell beginning of your life that would be an oversimplification, and since then the universe has poured far, far, far more "bits of information" through you than you could possibly have integrated into yourself. You would require a great deal more information to describe than your raw genome (just as identical twins are still, you know, distinct people). But in terms of asking what can be in the DNA alone, the information theory is relevant, and citing "interactions" buys nothing. The DNA has the bits the DNA has and it doesn't get any more by "interactions".
The organism with the DNA will interact with its environment, of course, and it's essentially proved that a lot of our "innate knowledge" requires interacting with the environment to develop. In terms of understanding the whole process, that will all be critical to understand. But in terms of literally what is in the DNA, that's not particularly relevant.
(My personal belief is that a non-trivial component of the confusion is indeed that we look at the "final product" of this innate knowledge and over-attribute it to DNA. Babies naturally recognize faces... except they don't. "Face recognition" is a huge amount of bits. On the other hand, encoding some instructions that will result in embedding an eigenface [1] into the visual system and giving a hint of dopamine when it recognizes one, and leaving the baby to eventually work out what "faces" are and why they are interesting isn't that many bits at all. The end result is "face recognition" and that system would consist of a huge number of bits' worth of information, but the vast bulk would be from the environment. What was actually in the DNA is just a tiny seed crystal of information, which fits perfectly comfortably in the DNA. Especially since it wouldn't even necessarily be a "thing that matches faces" but just a weird streak through the hyperspace of possible visual inputs that happen to privilege faces but will also privilege and disfavor any number of other inputs too, which reduces the requisite bits even further since we don't have to "spend bits" on truly selecting on "faces". A lot of our "innate knowledge" doesn't really take that many "bits" in the DNA.)
[1]: https://en.wikipedia.org/wiki/Eigenface - note even the samples the Wiki gives are more complicated and information-rich than would be necessary for the scheme I describe to work.
That's true for any encoding of information. KC can only be defined relative to some computational model, and the actual number depends on the details of that model. It just so happens that there is a large family of models whose KCs are equal to within a multiplicative constant. But the actual number is never meaningful in isolation.
So is the answer that we have no clue? Or that it's a meaningless question? Given that we don't know the computational model, I mean.
To me the interesting question is how much life memory could be encoded in the genome. Maybe top points from human evolution? Or nothing more than really simple basic stuff?
Leaving aside how it might get stored in the germline, and later accessed.
Both. The fundamental problem here is that the phrase "innate knowledge" hasn't been defined well enough to allow one to quantify it. How do you measure "innate knowledge"? In what units would you express the results?
Here is an analogy: consider movies. A movie is the result of a collaboration between many people: writers, directors, actors, editors, stunt people, special-effects folks... how would you go about measuring how much of the quality of the final product is due to each individual contribution? It's a very similar situation with the nature-vs-nurture question (which is really what this is). The sum total of what we are is a result of very complex interactions/collaborations between our genes and the environment in which they exist. Quantifying the contribution of our genes is not that far removed from quantifying the contribution of an individual on a film production team.
Answering these questions is not impossible in principle. It's just really, really hard, and at the moment we aren't anywhere close to being able to do so.
Useful knowledge might include sucking on nipples, being afraid of snakes and spiders, being wary of fire, sexual drives after puberty, and so on.
As far as units, why not bits?
> Here is an analogy: consider movies.
I agree that going from a zygote to an adult animal is a complicated process. There's the genome, the process of development, and a complex mixture of sensory data and actions. And yes, that doesn't do justice to the complexity.
But the question here is how much innate data. Not how substantial a role that data plays. So as a first approximation, wouldn't that be the script? Or maybe the script plus the video file?
> It's a very similar situation with the nature-vs-nurture question (which is really what this is).
I wasn't seeing it that way. There's no question that the environment plays a crucial role in development. But I don't see how any of that can create knowledge of past events. Which is what "innate knowledge" means to me.
Because bits are a unit of information, not knowledge. Not the same thing.
But even if you're really interested in information and not knowledge that's still a problem. You can easily measure the information content of the human genome because we know how it's encoded. It's much harder to measure the information content of the human brain because we don't yet have a complete grip on how it works.
> I don't see how any of that can create knowledge of past events.
That depends on what you mean by "past events". If you really mean individual one-off events, there is no known mechanism for that kind of information to get encoded in a genome. If you mean general information about the world acquired through "experience" (where having natural selection act on your population counts as "experience"), that could be encoded in a combination of information that resides in the genome and information that resides in the environment, where the structure of the brain is part of the environment (of the genome). We just don't know.
How can knowledge not be a subset of information?
"I am wearing a white hat."
Is that information also knowledge? Well, it depends. It depends, for example, on whether or not I am in point of fact wearing a white hat, because if I'm not then you cannot possibly know that I am wearing a white hat despite the fact that I have just given you that information. That information may or may not correspond to reality, and one of the things that distinguishes knowledge from information is that knowledge has to correspond to reality in order to qualify as knowledge, and information need not correspond to reality ("false knowledge" is an oxymoron but "false information" is not.)
But there's more to the distinction between knowledge and information than mere correspondence to reality. If I am in point of fact not wearing a white hat, the information above does not magically transform itself into knowledge if I suddenly put on a white hat.
So the answer to TFA's question is "It depends".
Being amazed that you can make a human with only 20,000 genes is thus like being amazed that Shakespeare could write all those plays with only 26 letters. It’s totally missing where the actual, meaningful information is and how it is decoded.
The simplified cartoon image that is associated with DNA, where phenotypes are directly connecting to specific genes - i.e. this mutation makes you smart, or that mutation causes cancer - is not at all how biology works.
The sentence makes exactly the error that it accuses others of making, the '20,000 genes' is not the encoding, that's the codons of which there are 64 some of which encode the same amino acid. The 20,000 really is like a single play of Shakespeare.
The quote is a criticism of a common way of thinking in biology, where many (if not most) practitioners think in terms "one gene does one thing" (or perhaps few things at most).
When one follows that mentality it becomes hard to explain how just 20K genes could make a living organism.
The genome must encode all the components used to make an organism, as well as the instructions on how and when to use these components.
The 'one gene, one thing' or just a few things at most meme died long ago for all biology practitioners that are with the times, the problem is more with IT people attempting to understand biology trying to make computing analogies to make sense of something that evolved rather than that it was designed.
Shannon information provides a trivial upper bound on Kolmogorov complexity: imagine a tiny program with a huge data segment, which just copies the contents of the data segment to standard output. In the case of the human genome, this value is on the order of 6 billion bits. The key takeaway is that a much larger fraction of these bits than one might expect from the phrase "20000 genes" are potentially relevant, not that Shannon information somehow underestimates complexity.
It seems likely that the amount of information in the DNA "decompressor" (in mammals, the mother and her environment) dwarfs the amount actually present in the DNA itself.
Thinking in those terms naturally leads to the kind of “counting arguments” that Bengio makes. These seem to take each gene as a bit of information, and ask whether there are enough such bits to specify all the bits in the brain, usually taken as the number of connections. Obviously the answer is there are not enough such bits. (There aren’t even enough bits if you take individual bases of the genome as your units of information).
As evidence for this we know there are all sorts of heritable epigenetic effects. It would also partially explain the partial failure of GWAS to pinpoint all an individual's features as was previously hoped before the human genome project completed.
It seems like there's a process called Somatic Cell Nuclear Transfer that might make possible to evaluate aspects of this, but it doesn't appear to have been performed sufficiently many times with different individuals to draw major conclusions.
However in reality, the gene dose makes a difference - so it’s more like the genome (+genome architecture) encodes for capacity to produce different amounts of A and B. So, with high A expression you will see things like “AAAAB”,”AAAAAA” etc.
The point is that rather than encoding for different functions, the genome encodes for the diversity of different message queues - and it is the length of these queues and rate of message firing that encodes (as much as you can in a probabilistic sense) for our cell programming. The messages could be “build a transcription factor” or “put a receptor on the surface”, or “make a crank that increases the capacity of these queues”.
If we are looking for the innate information, we should be asking the question: In how much information can we store the computer, and all machinery that builds it from basic components (molecules basically). It does become easy then to imagine that there is some kind of suckling response, a computer also starts doing things when it is powered up.
> You don’t need to specify where every synapse is to get the machine to work right. You just need to specify roughly the numbers of different cell types, their relative positions, the other types of neurons they tend to connect to, etc. The job of building the brain is accomplished statistically, and, crucially, probabilistically. This is why there is lots of variation in brain structure and function even between monozygotic twins and why intrinsic developmental variation is such a crucial (and overlooked) source of differences in people’s psychology and behaviour (the subject of my book Innate).
One of the interesting things that we've learned over the years, painfully and repeatedly, is that fairly simple encodings of fairly simple data can repeatedly generate extremely complex visible outcomes. A great example of this is patterning in animal fur- there isn't a section that encodes a pixel map of the resulting spots or their X/Y locations. INstead a collection of genes operate in a feedback loop to generate the patterns. Turing's theory of morphogenesis has turned out to be highly prescient application of very simply theories that does a good job of explaining observed patterning.
What I've seen repeatedly over the years is that the proteins themselves aren't that interesting- all the interesting complex repeatable phenotypes come from careful regulation of the expression of proteins with feedback loops and spatial gradients. The best work on this has been done in evo-devo field on model organisms (not humans). C Elegans was an exceptionally good choice for a model organism for many reasons, not the least of which is that every instead of the organism has exactly the same cells in exactly the same location, and that appears to be completely genetically encoded (again, not like a pixel map, but as generative rules).
It sort of feels like training a neural network is like rewiring an adult brain which has already fully grown. Are there neural architectures which grow over time, or "age", emulating the early development of the brain?
One form of this is culture and species taught behaviors.
have labs successfully changed these? I'm familiar with a prenatal brain surgery that made chickens behave like ducks, but it wasn't genetic
As for reflexes, don't we kow how they work on an anatomical/neural level in a grown adult? From my high school biology lessions, I remember that for certain reflexes, there are nerves that directly connect "sensors" to motor neurons in the spinal cord, bypassing the brain altogether.
If you stick to the "genes as a program" metaphor, "encoding" this reflex in the genome would mean that the genome contains the correct triggers and "subroutines" (i.e. as cascades of gene expression) so that a particular group of cells will form that nerve and connect it to the correct other neurons.
Otherwise excellent, if short, article.