Hacker News Reader

eh... Information theory wise, at best you can say you've compressed the knowledge to the action to the steps required to construct the geometry. But you've still encoded it on some level. So there is an information-theoretical limit that is still worth thinking about. I think they kind of miss the boat on their discussion of Shannon. You've encoded the structure, and the actual behavior then arises from actually running the network. I think they got a little confused to be honest.

And indeed, what people will find is exactly as you say: The DNA encodes how to grow the network such that it will respond with the evolutionary advantageous instinct.

I don't know if I'd quite say that no one is looking into that. Obviously we know that there are NN that use deep networks. But also we can see that we can increase the computation power by removing nodes and edges that don't contribute to the correct answer. In evolution, this is a literal cost savings in that you need to feed less neurons.

I understand your point, and I'm having trouble articulating this thought, but I think with inductive bias you can structure your net such that it is primed to naturally learn certain behaviors that are more complex than the Shannon entropy required in the encoding DNA. The minimal structural information combined with environmental interaction can produce far richer information content, past the limit of the original DNA, but from a design perspective your actual code is very compact because you gain the rest of the knowledge required for innate behaviors naturally from the environment.

It's interesting to think about all this. Everything DNA encodes is responses to the environment.

On the molecular level, proteins are only functional at certain temperatures, pressures, and pH levels.

On the cell level, certain proteins might only be found in specific areas, and essential amino acids might be harvested from the environment rather than produced by the cell itself.

On the organism level, we are predisposed to shiver if it's too cold or sweat if it's too hot, an uncomfortable feeling, so innately we go to where the climate is milder and won't kill us. We clothe ourselves and that range is expanded, suddenly there is a selective advantage for being able to clothe yourself, just like there is to be a white beetle on a white sand beach instead of a green one visible to predators, or a cell that's able to just mooch someone elses amino acids rather than expend energy making their own.

As the article points out, Shannon information really is the wrong benchmark. Kolmogorov complexity and the busy beaver sequence should demonstrate that there's very little reason to argue for any sort of genetic bottleneck on gene count grounds.

"So there is an information-theoretical limit that is still worth thinking about"

So, technically, sure, but doesn't the busy beaver sequence imply the potential of anything more complicated than a 5 state turing machine is literally unimaginable? I mean, unimaginably huge even as a multiple of the size of the universe.

https://en.wikipedia.org/wiki/Busy_beaver

I think there is perhaps a dozen reasons why this is an incorrect comparison.

If we want to measure the information limit of DNA, we can argue the DNA is the program input... sure, so far so good. But then what is the "tape"? The tape must be empty at the beginning of the program... So, I guess we really need to consider the "program" is the entire universe of atoms...

Ok... but then... who said that the program ever Halts, exactly? Busy beaver needs to Halt.

So, we actually broke two of the fundamental assumptions of the Busy Beaver problem. So I wouldn't make any guarantees about the complexity of DNA based on it.

Inductive biases are not a new topic in ML. Also your examples have little to do with inductive biases as inductive biases are about generalization on out-of-sample data through human based restriction of the model hypothesis space.

> Point being, you don't need to encode, say, the image of a nipple and the action of suckling in DNA to create innate suckling knowledge - you simply need to structure the brain such that it is biased to perform and learn certain sequences of behaviors.

How do you think "structuring the brain such that it performs suckling" differs from "encoding the action of suckling"?

You just said the same thing two different ways.

No, not quite. This is a little hard to describe outside of ML terminology and it's pretty new, but I'll try.

The structure I'm describing is at a different scale from the structures of the individual neurons. Actually, come to think of it, baby motion may be an even better example:

When a baby is born, it (possibly) has no understanding of how to move it's arms and legs. But the newborn brain is structured in such a way as to encourage movement of limbs, such that the movement required for learning becomes a high probability event°. In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment.

If you connect neurons in a particular large scale pattern, and wire some up to ears, some to eyes, some to touch receptors, two things may happen:

1. Your brain is more likely to perform the initial actions required to learn

2. Your brain may more easily learn certain actions

The second thing is inductive bias, and I think it will be extremely powerful in the world of ML once we figure out how to harness it. We know that neural nets are hugely wasteful now, there have been papers published on neural network optimizers which can shrink nets by some 80% while maintaining accuracy. Inductive bias is probably such an optimization.

> In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment.

That is all of the information necessary for grasping. By your own stipulation, it is sufficient to enable grasping. Nothing more can be necessary.

What you're talking about is the degree to which the information is compressed at rest. But index.html and index.html.gz don't contain different information; they contain the same information encoded differently.

No, it's not compression at rest. What GP is talking about is the encoding of recipes for acquiring skills rather than encoding the skill itself. The end result is that the skill is acquired, but the skill is not latent in the genome, so your .gz analogy doesn't hold. It's a compression that has access to a vast existing dictionary and makes better use of it then some other.

The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. This is what one would expect given what we already know about biology and Turing machines. The difference is the Turing machine here doesn't start with an empty tape -- far from it.

> The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome.

Why not?

Consider birdsong. Many birds must learn "correct" song from models. If raised without good models, they still sing, but they sing a deformed song. This is, correctly, treated as evidence that the song of these birds is a cultural phenomenon -- a bird raised in isolation will sing in an identifiably different way than a bird raised normally.

So is normal song arbitrary? Can we say that it isn't encoded in the birds' genetics? Those answers are no and no; when birds with deformed song raise children, those children learn normal song. Normal song is the only steady state of the bird population; any other state will inevitably become normal song.

This is sufficient to say that normal song is latent in the birds' genome -- the genome will create it from nothing if necessary -- even though the genome cannot cause a single individual bird to sing correctly in isolation.

That's just one, particularly limited, way of compressing information by using the environment. The information is merely a fixed point of the "function" that is encoded. But what about all the other sources of signal that don't originate from the organism's own genome? If there are other ways to compress the information even further, should we not expect Nature to use all of them?

Consider human language. Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not.

(Strictly speaking, the genome can't even create the bird from nothing, but I understand what you are getting at.)

> Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not.

Which? I would say the answer is clearly yes. The closest analogue we have is the innovation of sign language by deaf children who were brought into schools with each other. They created working sign language from -- again -- nothing; they could not learn spoken language due to being deaf, and they had no sign language model to learn from. This took only a single generation, even better than the birds.

https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language

This is an exact match to your hypothetical question if you're willing to overlook the fact that sign languages are not "spoken".

The hypothetical would be, as a thought experiment, what if Mars was colonized by a bunch of newborn human babies? Leaving aside the question of how they'd survive, would they develop languages like the ones we have today within a few generations? I think the very idea is ridiculous, but you may disagree.

Nicaraguan sign language was not developed in a vacuum, it was developed by children living in a completely normal human society with the full use of all their senses but one. Far from "nothing".

It was developed by children living in a completely normal human society with zero exposure to language. So yes, ignoring the question of how they survive, a community of children will develop its own language without the need for external input.

Why do you think differently? "I think the very idea is ridiculous" isn't much of an argument.

> It was developed by children living in a completely normal human society with zero exposure to language.

If they are living in a normal society, they are exposed to language nearly continuously, from visible printed language to visible body language to the visual evidence of language being spoken all around them.