Hacker News Reader

Apologies for the vagueness there. Also apologies for the lack of structure following. I don't have time to write an essay, so I hope you can settle for some meandering commentary.

While the more complex topologies, activation functions, training algorithms are interesting, the current workhorse of ANNs are still plain MLPs (to my knowledge). So, yeah, I was referring mostly to MLPs.

The argument I'm doing a pale job voicing is the one that exists between the statistics and ML communities. For those unfamiliar, stat people generally demand comprehensible, general statements about the performance and meaning of the various techniques. This has historically largely restricted the power of statistical techniques. Advances in computing expanded in the ML field because people aimed to solve simpler goals (maximize training and testing predictive accuracy, ignoring why or how it works). This was a major force in the current renaissance in learning technologies, but leaves a big divided camp.

One side demands theory, story, and proof. The other just demands results. Which is great. They complement each other.

---

ANNs (MLPs in particular) bother me because they're essentially general non-linear discrimination boundary learners. Once you have that decision boundary you can (with the usual enormous concerns toward overfitting) predict new data with arbitrary accuracy given enough compute power, large enough layers, and enough training data. But even with all of those things you won't learn much from your parameter space.

Compare this to, for instance, LDA. It's hard to be certain that the usual high-level interpretation of LDA in data mining (that it finds mixtures of topics on documents) is terribly meaningful, but you still are able to learn a lot about the data space by examining the topic space. It's the kind of thing that allows for many interaction points between the algorithm and the users.

MLPs of course will also induce a latent representation on the hidden layer. There are really fascinating implementations of non-linear PCA that take this approach, but it's not clear to me what the properties of this latent representation are or how to influence them.

---

In some sense, the heart of my argument comes from how flexibility can hurt you in a learning algorithm: it makes your parameter space less meaningful (though I imagine mutable topologies can help this a lot — I have a little experience doing grid search over recurrent nodes and attempting to rank signals by their temporal information using the resultant best-performing topology and I'm sure that rabbit hole goes far deeper).

I'm not looking to be awed by a rule based system or GAI (whatever the hell that even means). I want hypothesis spaces which allow you to flexibly describe a question about a large class of data and then learn things from the optimum. I also want these algorithms to solve real world problems. Bayesian modeling gives you a lot of the first one. MLPs give you a lot of the second.

> I guess this depends on how advanced your toolkit is.

I'm assuming you mean this to counter the "simplest of biological neural networks" part, rather than the "completely opaque black boxes" part. Nonetheless, I think it is fair to say that ANNs still fall far short of the complexity of even the simplest biological brains. For example, even the nervous system of a Hydra features neurotransmitters [1] (as opposed to just heterogeneous signals of ANNs) and brains are affected by their own electric fields [2].

> You want to be able to discern some rule base from the system that you can understand? Even biological brains do not have that property.

More generally, transparency is important to be able to understand the inductive bias you've built up, and to try to alter or refine it in useful ways.

Also, the fact that our brains lack transparency doesn't justify leaving it out of an AI system, nor does it demonstrate the difficulty of building a transparent AI -- nature just had no drive towards transparency. Plus we (humans) can introspect and explain our reasoning.

> See, but the problem is that we don't even have the knowledge of how real brains work to start forming such principals > based on a technique which reflects the structure of the problem of learning, generalization, and hypothesis search and illuminates it.

This is only the case if we're speaking about biological brains, and not as a generic word to mean "intelligent system". In the latter case, we do in fact, have quite a bit of knowledge about such principles from reasoning about hypothesis spaces. From where we get things like active learning, or Solomonoff's work on Universal Induction [3].

> It's not going to be a fully general, "human-level" AI, obviously, as that requires huge amounts of semantic knowledge about the world that was encoded through billions of years of evolution

By "semantic knowledge" do you mean inductive bias? Because otherwise I'm at a loss. I don't believe that the picture for Artificial General Intelligence is as bleak as you make it sound though.

As an aside, you mention "Lots of more biologically-inspired approaches have been explored". Do you know of any projects looking to mine the structure of various parts of brains to figure out the sorts of inductive bias those structures correspond to? (As opposed to just copying structure) What I mean, is that presumably if a part of the brain heavily involved in recognizing faces has a unique structure/wiring, that structure is optimized such that it performs well on face recognition -- and correspondingly poorly on something else, per no free lunch -- and that optimization should tell us something about the nature of recognizing faces. Sort of in the same way that the use of a Naive Bayesian classifier rather than a Bayesian Net might tell you that the classifier is optimized for cases where the variables are independent.

[1] http://books.google.com/books?id=WWN_t498S5IC&pg=PA12...

[2] http://www.scientificamerican.com/article.cfm?id=brain-elect...

[3] http://world.std.com/~rjs/pubs.html

Thank you for your solid reply. I think ANN is the real thing, and most of the other "AI" solutions that we have give the impression of intelligence, but are simply algorithms that solve problems we have in very well defined ways.

ANN is a general intelligence that gets shaped by experience and adapts figure out its own solutions. This is the path that will lead to our concept of AI presented in sci-fi and what the general public actually thinks of. It isn't some coder banging on the keyboard trying to replicate what a human would do under a specific situation, it is something that truly learns.

We have a long way to go before this becomes a reality, but it will happen.

Tansey wrote a good, solid reply, but I dislike worshipping at the altar of mysteriousness. When a human or better GAI finally gets built, it won't be entirely hand-coded (although the seed for it might be). But the better we understand how our AI works, the better we can make sure it's doing what we want it to; and not, say, crashing the stock market with millions of erroneous transactions carried out too fast for humans to correct.

That's very romantic, but I think instead we're going to learn that intelligence is not such a well-defined idea as to be engineered into anything imagined by science fiction.

Further, there are plenty of techniques which aren't ANNs but still are "shaped by experience" and "figure out their own solutions".

I side pretty strongly with the idea of building tools to bring the powerful modes of computerized learning close to the powerful modes of human learning. I think we're inordinately far away from replicating one with the other.

> I think ANN is the real thing, and most of the other "AI" solutions that we have give the impression of intelligence, but are simply algorithms that solve problems we have in very well defined ways...ANN is a general intelligence that gets shaped by experience and adapts figure out its own solutions...It isn't some coder banging on the keyboard trying to replicate what a human would do under a specific situation, it is something that truly learns.

It's not ANNs on the one hand and "some coder banging on the keyboard trying to replicate what a human would do" on the other.

Supervised learning is the most common use case for an ANN, and means learning from labelled training data during a training phase and then not changing; it can be done with a number of techniques, including Support Vector Machines, Decision Trees, Bayesian methods, and so on.

Reinforcement Learning on the other hand, is continuous learning from trying things and making mistakes -- experience, in other words. It too can be done a number of ways, like with Evolutionary Algorithms, Markov Decision Processes, Inductive Logic Programming, etc. Hell, PG's spam classifier learns from experience, and it's nothing more than a Naive Bayes classifier ( http://www.paulgraham.com/spam.html ).

Without going on for much longer, my point is that there really is no reason to exalt ANNs the way you have.

Disclaimer: I'm a huge believer in Artificial General Intelligence via Competent Program Search ( see http://metacog.org/research.html + http://www.google.com/url?sa=t&source=web&cd=2&v... + www.idsia.ch/~juergen/ )