Hacker News Reader

Comment by flangola7 | Hacker News Reader

Then how do humans do it? Our knowledge relies heavily on sending the output of humans to the input of humans (verbal communication, books, self-thought, etc). At one point total human knowledge was little more than "some berries are bad" but through this recursive process arrived at the rich garden of information we have today.

View on HN

Because humans (and your language is confusing here between individual humans as models and human society as the model) are constantly receiving new information. Even before humans learned "some berries are bad" they initially thought either "all known eaten berries are bad" or "all known eaten berries are good", then somebody at a good/bad berry and updated their believes (i.e. their model of the world).

Humans learn from other humans because humans don't individually share the same information and model of the world. A science teach can speed up how you learn science by taking the compressed information and explaining it quickly (essentially what is happening in the post), but out scientific model is expanded when we have experiences that call into question the strength of our current model.

However both individually and as a society we are constantly taking in new information (sometimes more sometimes less) and using that to update our model.

Humans don't just try to imitate text from some group of humans, like language models. But if we wanted to imitate the text of, say, 18th century novelists, then only those 18th century novels seem to matter. If we also started to imitate our own imitations of 18th century novels (synthetic data) the result would probably get worse, since our imitations are imperfect.

To consider your analogy a bit further: Imagine you had access to only... five 18th century novels, that's it, no other knowledge about the period, the people or anything. Then you try to write your own. You can make up your own stories that are a merging of various parts of those five novels, but they'll always only contain what's in those, or some combination.

If all the books deal with France then Germans or English will never make an appearance because it's literally impossible to guess that they exist since they're not in the original books (the books here are training data if that's not obvious).

Humans have nothing in common with neural networks (other than if you choose to misunderstand the literal definition of those two words), there's no direct analogy to be made.

I didn't say anything about neural networks, I asked how humans do it.

To start with, humans have access to far more data than just what is generated by other humans. We can look at nature, other animals, mathematical laws and many other things.

Humans also have far more ability than current model architectures to use memory (particularly in training), flexible context (Model context always directly follows the input distribution in training whereas we can flexibly combine different parts of our input data as we like ) and logic. We can also do live learning in ways current models can’t.

Reply on news.ycombinator.com