> Sorry if I got a little bit off topic there.
Not at all. Though forgive me for not agreeing with some key points you raise.
> Curiosity is not a necessity. While curiosity is integral part of what makes a human well, human, it doesn't have to be hardwired.
> The keyword here is cheap. Any sufficiently powerful maximizer with an infinite horizon __has__ to develop curiosity otherwise they will not be able to maximise their reward function.
With that I do in fact agree. I think curiosity was a quick solution to fix some immediate problems I was seeing from the pain-slash-survival angle, perhaps from a belief there should be more to humanity. I also feel it is rather emergent and should (must!) emerge pretty fast, even, in order to survive.
Actually typing out the last sentence made me realize another meta-meta-level of intelligence. Whereas the basic reward function is level 0, the chemistry surrounding and interactions with out body it might be level 1 (might be, because they are probably emergent as well), evolution is definitely level 2 (or 1) - the multi-simulations of agents. On top of that, there's the fact that initialisation is cheap: meaning that even if some emergent properties are highly necessary on top of the basic reward function, and might lead to very complex aspects later on, a designer (and this is a very badly chosen word mayhaps) would be prepared to deal with those given the fact that there are many chances. Many one-cell organism striving to do better in the "soup". The more I think about this, the more I start becoming convinced that computational biology should have been a serious field (and many are saying this).
> In-fact, I'd argue that this is true for most mammalian functions such as taking care of our pack, exhibiting pro-social behaviour and so on, but there is a caveat. For this to happen, there needs to be an actual benefit in the behaviour.
See, this is where I respectfully disagree. The benefit can be emerging from a longer-term simulation rather than immediately. You might say: sure but what are the chances of this happening? Well how many intelligent species like ours have we encountered so far? On this planet, in this universe?
> Due to evolution operating mostly linearly with small changes through genetic and epigenetic information passing, there seems to be relatively little variation between generations, which then implies that it is difficult for some candidates to overwhelm everyone else in a winner takes all fashion, hence maximization of replication will eventually result in cooperation simply because that allows genes in support of it to continue replicating, effectively self-selecting for itself.
Yes and no. From a gut feeling I agree though I also think small changes tend to take over very rapidly in a population pool once they show up. The waiting is mainly for the showing up part.
> We saw this in the OpenAI video where agents eventually learned to cooperate in what was effectively a prisoners dilemma. In the video, there were two teams, the hiders and the seekers, in an environment that could be manipulated.
I saw that as well, and this is why although I believe at some point this will be possible, the main reason why I agree with hotz is because our simulations suck and always allow for exploitation. Unless it doesn't, of course. The on-device part, hence, for me, is not that necessary. But it means we should have a very robust simulation (which so far we don't have in any area of RL and associated topics; digital twins are a joke; people making them care more about dataviz; and so on).
> Regardless, what causes us to learn is neurotransmitters getting released because certain circuits activate, the neurotransmitters charge a neuron which causes it to fire. Connections between neurons get reinforced if they are frequently used, which reinforces them, makes them cheaper, and that inevitably reinforces certain patterns of behaviour.
Too tired to go into this but you touch upon some key differences between gradient descent and biological neuron learning, though we are getting closer to that: spiking neurons, memory cells, even real-neuron cell chips. I am not sure electronics wouldn't be able to emulate it correctly in the end. If after all, P <> NP, then what does the "real" difference of a computational time step make?
> What i propose is that we should instead look into analogies as a means of learning. Humans seem to be great at using analogies. Mathematically speaking, an analogy is a functor between categories.
Agree, and surprising that this is still such an open question in AI even given the one-shot and zero-shot learning research. Though it seems like this has been put on the backburner yet again. It amazes me even today how young humans are so good at that. Like someone said: show a cartoon tomato and a real tomato to a toddler. Next time show a cartoon of an elephant and wait until they see a real elephant. They will shout: elephant. Though on the other hand, the solution for this might be very close to use. A small architectural or multi-modal change. I was more pessimistic about this a few years ago, but less sure today.
I think the main piece missing of the puzzle is stepping away from supervised learning, self-supervised learning, and going for a continuous self-supervised reinforcement learning, where predictions for t+1 are continuously matched with reality, like a human brain does. The only problem is that you need to have a continuous reality. But we have that.