Hacker News Reader

What confuses me about your argument is that you are treating a child as an individual without context. The human brain, however, has reviewed bajillions of bajillabytes of training data over the course of its evolution.

The brain is already wired to have certain types of perceptions, so by the time a baby is born that child is just looking for that entity that fits the mommy definition, not trying to define a that it will have a caregiver and the optimal strategy is to bond to that caregiver.

Anyway, there are lots of examples where what you are saying holds, kids drawing general conclusions from limited data, but we also have to remember that the human brain is not a blank slate, it is already highly specialized at the time of birth.

> The brain is already wired to have certain types of perceptions, so by the time a baby is born [...]

A structure that evolved over time, amusingly, essentially by exactly the kind of "trying stuff at random" algorithm the grandparent is complaining about.

I guess the current generation of neural nets also arose by trying stuff at random, then? I kind of thought the people who invented them, Geoff Hinton no less, would be prepared to admit they may have had to do a liiittle bit of thinking.

I believe his point isn't that no thinking was involved in creating neural nets, but that our own ability to think is the result of a long line of random adjustments to optimize the human body for survival, the implication being that giving an entity a goal and letting it attempt a wide range of things randomly to achieve that goal isn't that silly of a method.

Yes, I get that, but I think it is silly if you don't have a few billions of years and a few billions of species with a few trillions of individuals to optimise over (like evolution had). If you have fewer resources you have got to get smart and in fact, that's exactly how people work with algorithms: they try to avoid using brute force.

Well ish. Machine learning algorithms are cutting it very close to brute force most of the time.

Consider this... the genome is ~725MB, and 99% percent is shared with chimpanzees, leaving ~7.25 MB for human-specific traits. While I'm sure that the mother instinct is strong (and probably encoded in that 99%), it's truly amazing how little storage that is!

Edit: My point is not about imprinting (which I don't claim is restricted to humans). I'm claiming that what we consider "human-specific" occupies remarkably little genetic memory. To understand how human language acquisition works with so little genetic context, we need to study humans and not NLP algorithms.

Chimps also have the mother instinct, as do most other animals.

Well, imprinting specifically is likely much, much older than the chimp/homo divide.

It also takes us about two years to learn to walk and say stuff that makes sense. The devil knows how many bajillabytes of information a human brain inputs over the course of two years.

The input is quantifiable. Our eyes see Less than a gigabyte / sec. The other senses even less. I would say a baby processes less than a petabyte of sensory data in a year. We are approaching computability of lifetimes of sensory data...

A human child receives millions of "training" lessons every day for two years, and only then does it learn that you, indeed, are its "mother".

This knowledge is composed from other, simpler and more basic facts, that it slowly learns along the way:

  1) you are a reliable source of warmth and comfort
  2) you make me feel safe when I'm anxious and scared
  3) you feed me when I'm hungry
  4) you always come back after going away
  5) you make me laugh and feel happy

These can be broken down into more basic facts, as the concepts here are still rather advanced (for a baby). The baby doesn't actually know what the it means to feel "safe", "fear", "happy", etc. It just has fuzzy experiential feelings that slowly become sharper and more contextualized as the baby is "trained" every second of every day by interacting with the world around it.

This strikes me as highly speculative. If only we could ask a one year old, but we can't. I personally find it pretty suspect that it builds a definition of a mother from disparate logical components.

I think I didn't properly communicate my thoughts in the first post. Let me try to write them out more clearly.

I wouldn't describe them as disparate logical components. The numbered points I wrote are the abstract, rough, adult-level interpretations of the messy "knowledge" that the baby develops. The baby doesn't have this type of vocabulary or even scope of understanding. But I think those would be the flavor of the pillars of knowledge the baby builds up to develop a conception of "mother".

The 2-year old's conception, when they first speak "Mama", will still be incredibly rough. It will be a faint glimmer of what we actually mean when we use the word as fully grown adults. But each additional day more understanding is built up as the network of neurons and their activation patterns, etc is being modified through further interaction. The experiential data from every additional moment of time that passes are pattern matched, integrated, and added to what it already "knows". Humans are really, really good at loose pattern matching & loose analogies. It feels to me as if most of how we think and what we know is just many, many layers of this.

Side point: I would say that even two adults never mean the exact same thing when they use the same word, as they have built an understanding of that word/concept from different life experiences. We can try to communicate to others what we think a certain word/concept means to us, however we have to use other words/concepts to do that, which makes the problem a little difficult! This is why people very often talk right past each other.

You are 100% correct, this is a lot of speculation. Hopefully we will make more progress towards knowing the hard science of how it actually works in our lifetime =)

Interestingly, from your list most of those are (also) applicable to a father. A baby can nearly immediately after birth also recognise its mother by smell and sound.

The baby is learning in the womb too, I would imagine, so it should recognize the mother's smell and sound.

There is a very good reason for that which is explained very eloquently in this speech. Really worth a look.

https://youtu.be/dXQPL9GooyI

"You cannot find things by looking for them"

Any person who have ever tried to solve problems (not just rationalize next steps based on previous findings) will know that there is really something unfruitful and irreproducible about logical reasoning.

One of the primary reasons is that logical reasoning you make is based on it's premise. As an example you can rationalize your way through Game of Thrones as long as your premise is set to accept a number of sub-conclusions. Or be a theologist rationalizing your way through the bible.

But this powerful process is also a cage which keeps you trapped inside the same logical consequences of your premises.

And so to invent something new you have to be "irrational" to break out of the boundaries of the premise.

So the claim is not to abandon logic but the belief that problems are solved and inventions made by rational logical process alone. It turns out that most things are actually more trial and error which again also historically has some merit.

>> One of the primary reasons is that logical reasoning you make is based on it's premise.

Yes, that's called the closed-world assumption. Something is true if you can prove it from the things you know.

You know how else we call the "things you know"? Data.

You know what else we do with data? Train machine learning algorithms.

You know where machine learning algorithms would be if they abandoned their (implicit) closed-world assumption?

Where machines would be if they increased the available input sources? We know that as little as we know where a kid will end up when we put them into the world and they start to interact with it being the patter recognizing feedback loops that we are.

The point is that premises allow us to make logical reasoning but the premises themselves are not necessarily logic or universally true or objective or any other notion of right or wrong.

Look through history. Most of the really big innovations weren't a product of some objective but rather incidents and curious phenomena we never expected.

Rational reasoning will take you some of the way as long as you have the premise, but the question is how to get to the premise.

If a machine learning algorithm takes closed-world assumptions too far, it overfits the input data. We test whether it's working by testing it on data it hasn't seen.

You can't tell whether overfitting is happening by just looking at the input data. You'd at least need a holdout set for testing.

Unfortunately, the article is down right now, so I don't have the context for that quote.

But in "Thinking Fast and Slow", Daniel Kahneman makes that same point that most of our reasoning is kind of heuristic pattern matching. There are lots of experiments that show evidence of this behavior cited in his book. Certainly people can turn it off and logically reason about things when desired, but it's slow and taxing, and the heuristics people live by mostly work, so it's frequently unneeded.

Logical reasoning is a tool we use to form an understand the world, but it is not the foundation of the understanding itself. To use an analogy, saying humans are fundamentally rational because we use reasoning to make decisions is like saying birds are musicians because they communicate with song.

The article was absolutely not saying that we should abandon logic. It simply said that our efforts in building any sort of "understanding" with computers have been more fruitful when relying on analogy and classification as a basis rather than "pure" logic.

Classification of course, assigning entities to categories according to their attributes, is itself logic.

Making a distinction between "reasoning by analogy" and "reasoning with logic" is actually pretty weird. Analogy is also (informal) logic - drawing attention to the common characteristics of objects and reasoning about their therefore shared attributes. What Hinton really means, at least as I understand it, is "reasoning with approximation" not by analogy.

But, approximation is what you do when you _can't_ do logic. It's your recourse when you can't do the optimal, which is indeed, reasoning with logic according to knowledge.

Hinton is advocating that, because we managed to get ahead with some hard problems by approximating solutions, we should forget about ever getting a good understanding of the problems. But that's dangerous, defeatist and goes against the way we've progressed pretty much throughout all of human civilisation so far.

Analogy is pretty logical though, you are essentially casting to something and seeing if you can manipulate it in a certain way.

This casting part is absolutely illogical :) And there's yet more obscure part in it: find what to cast to. But once you did this, you can return back to logic.

I don't understand where you see that argument in the article.

The statement was descriptive, not prescriptive. The claim is that the core of our thinking is analogy-based. It doesn't say we are incapable of logical reasoning, or that it is undesirable.

If one agrees with that descriptive statement of human reasoning, then to make an artificial system that does similar things one should have it also reason by analogy rather than logically at its base.

>> The statement was descriptive, not prescriptive.

It's a statement of opinion that declares a strongly held view with the clear intent to influence the views of others.

Frex, when theists say that God forbids premarital sex, they don't only say it because they believe it to be true, they say it because they wish you to adhere to their own beliefs.

That isn't clear to me. Computers are good at arithmetic and sequential algorithms by design, but those are also applications of their power, and categorizing them by the intended application is also pretty dumb. It is the tricks and happy accidents that propel a lot of innovation - indeed, the fact that such has happened is one the numerous challenges to positivism.

I wouldn't dismiss logical reasoning this way either. But i think we are underestimating the power of analogies. Every abstraction is an analogy. A lot of higher math is generalizations (or recursive applications) of real-world , easy to understand analogies. It would be more fair to say that logical reasoning is a result of combining analogies.

Would it be easier to swallow if it were called clustering instead of analogy? Simply put, logic doesn't scale. Unless you've already solved language, you can't have sufficiently detailed and accurate logic to be useful. But clustering does scale, at least this limited version of clustering.

>> logic doesn't scale

You have to try and remember that our current computer circuits are (boolean) logic, implemented in hardware.

In fact, there's a very good reason why that is so. Turing, and the logicians of Russle's and Goedel's generation before him, conceived of computers as logic machines that reasoned like humans. In Turing's time, computers with logic circuits were even proposed as artificial intelligence. And you know what? For their time, they certainly were.

Yet, somehow, a couple of generations later we're ready to do without logic. Except, that's what our computers run on. But who knows, maybe quantum computers will run not on logic circuits but on hardware neural net implementations.

It will take petabytes to train a "helo world" program and you'll have to wait for a couple of hours for it to learn, but it will be worth the pain, just to get rid of that blasted old logic that "doesn't scale".

Our computers run on transistors being turned on or off. Thats the premise that creates the rest and allow us to use it for boolean logical circuits which again allow os to simulate almost anything.

The logical part is our interpretation of phenomena not necessarily "out there".

We are pattern recognizing feedback loops the rest is interpretation IMO.

In this context, I'm referring to logic as a knowledge base or some similar. Either this information has to be manually curated (which doesn't scale), have huge gaps, or have loads of false positives (ie incorrect information).

No one is arguing whether we should or shouldn't be using programming languages or hardware based in logic.

Alright, thank you for clarifying this. I guess I got carried away a bit last night and allowed this to escalate to a sort of unproductive us-vs-them kind of thing. Sorry about that. It wasn't my intention and some of my replies from yesterday are a bit shit.

Now-

>> this information has to be manually curated (which doesn't scale)

Aye, that's exactly right. I know this first-hand. I study machine learning, so I'm not really-really against it (it works, so). My hunch though is that although we've made a lot of progress with what Hinton calls "analogy" (and what I think he means to call "approximation") we haven't gone _that_ far and every time we made some progress we've eventually hit a wall.

Take language for instance: first there was the Bag of Words model, language as independent words. That worked for a while, but eventually it hit a wall. Now we got word embeddings. It gives great results and yes, you can do arithmetic with words, which is major, but then we still haven't made real progress.

And note that the top-rated NLP system currently, Watson (Schoher's grad student obviously didn't make something as performant as that) combines statistical learning with knowledge engineering. They use a statistical parser to fill in frame [1] slots. Each approach handles different aspects of the problem. Both together cover more of the solution than each on its own can ever hope to. If you want to do really well, you need to pick the best of both worlds, approximation and knowledge, together.

[1] https://en.wikipedia.org/wiki/Frame_%28artificial_intelligen...

>> clustering does scale

I was reading Lin & Wu [1] today; have a look in there for this bit:

to create 3000 clusters among 20 million phrases using 3-word windows, each K-Means iteration takes about 20 minutes on 1000 CPUs. Without using the indexing technique in Section 2.4, each iteration takes about 4 times as long.

So, sure, clustering scales... if you have a um, cluster, of 1000 CPUs lying around to spread it over.

You're saying something fundamentally wrong here. There are classically efficient algorithms (linear and sub-linear in time) for first order logic with search, as in unification, RETE etc, whereas all the machine learning algorithms are typically starting at quadratic times and then increase in crazy, mad, insane rates of complexity that would never be accepted in any other field.

It's machine learning that "doesn't scale", not logic.

[1] http://www.aclweb.org/anthology/P09-1116

It's not that we should abandon logic. But logic alone is not sufficient. It's very powerful but has a weak spot: it cannot create new ideas. We have to do this illogically. But once we have an idea, we can use all the power of logic to explore it. Here both parts are essential.