Is There a Smarter Path to Artificial Intelligence? Some Experts Hope So
nytimes.com
nytimes.com
The currently trendy "AI" looks more like massive data mining with powerful ML to me, that's very good for certain tasks but brings us nowhere near real AI. The knowledge representation problem has not yet been solved. In order to get even just a convincing simulation of AI, let alone real AI, we need a large common sense knowledge base / computational ontology.
One perhaps promising approach are geometric meaning theories/concept representations that allow for logical combinations. For example, Diederik Aerts works in this area. However, to be honest, I don't have the math skills to evaluate his approach. Generally speaking, logical modelling is too limited for good concept representations - especially classical 90s AI like in default reasoning and other nonmonotonic logics -, whereas traditionally geometric representations suffer from problems with representing logical inference and quantification. IMHO, that's a problem worth looking at. (Admittedly, I'm a bit biased towards symbolic AI like the people in this article.)
In principle, First Order Logic and equivalent languages can represent anything that can be represented in natural language. The problem is that in practice it is very hard to transfer into a logic language all the knowledge you might need for a useful system.
In fact, this was one motivation for at least one branch of machine learning- the work from Ryszard Michalski, Claude Sammut, Ranan Banerji, Steven Vere, Brian Cohen and later rule-learners like Quinlan's decision tree learners and of course all the later work in Inductive Logic Programming by Plotkin, Shapiro, Muggleton etc. These are all rule-learning systems, that can avoid the knowledge acquisition bottleneck that probably did in for purely rule-based expert systems in the '90s (i.e. getting experts to transfer their knowledge into rules).
I rather agree that further progress in AI will require a, let's say, synchretistic approach- like I say in another comment, the obvious thing for me is to use deep learning for perception, logic for inference and probabilistic modelling to deal with the noisy world. There are some people working in that sort of direction, with different amounts of emphasis on each of the three approaches. For instance, Josh Tenenbaum at MIT, Evans and Grefenstette at DeepMind, Luc De Raedt at KU Leuven, Kristian Kersting at Dortmund, Lise Getoor also at MIT, Pedro Domingos at Washington, and many others.
Apologies for all the name-dropping without links. Let me know and I can provide them if required.
It seems more likely that First Order Logic can be represented by a subset of natural language.
Otherwise, I agree. I don't understand why everyone has been concentrating so hard on probabilistic ML when there are many applications where inductive inference is more efficient and powerful.
Is there any proof about an existing non-leaky abstraction between natural language and First Order Logic (or logic in general)? Afaik there is none, but I might me wrong. For example (one amongst thousands): “I don’t like you” can actually mean (and it often does) the exact opposite, and the same can be said of a simple statement like “I am ok” which also can mean the exact opposite. Natural language is a very hard problem.
Notice again that I don't think this is feasible in practice. At least not with hand-crafted FOL expressions and certainly not with existing technology.
I'm not sure what "non-leaky abstraction" means. Would formal language theory fit the bill?
https://en.m.wikipedia.org/wiki/Heterophenomenology
In this framework, utterances can be studied without taking their truth value at face value.
I mean, I don't believe that is true any more that I believe calculus, or linear algebra are the way we think. Those are just formal systems that we can use to get various results about the world- but is that how we really think inside our heads? I kind of doubt it.
I'm just talking about the expressive power of first-order predicate calculus here, not about whether it's a good model for human intelligence.
I doubt this is the case since natural language makes a lot of use of metaphors, and our concepts are probably mostly based on the types of bodies we have and how we interact with the world. See embodied cognition and writings of Lakoff and Mark Johnson.
There is ambiguity in natural languages so the determination of underlying meaning is at best probabilistic, but we have ways of handling that too.
In fact, populating a LS/CS knowledgebase with requisite facts was a game stopper when they were first invented and still pose challenges of scale that remain unsolved decades later. If it accomplished anything, Cyc proved that manual curation alone of a rich database of propositions is depressingly infeasible. So how does one automatically curate a general fact KB, no matter what semantic model the facts take? I don't think expressiveness of the propositional representation will ever decide that question.
I think this tradeoff in quality vs quantity has been central to the inability of most Good Old Fashioned AI (GOFAI) formal symbolic methods to 1) scale up beyond toy academic clean-room problems (from the admittedly little I've seen of commercial LS/CS), as well as 2) support the flexibility and imprecision needed to deal with a messy ill-defined very big world with a seemingly unknowable number of unknowns.
It may be that this venerable yin v yang of knowledge modeling will forever bedevil AI. I don't have the sense that any techniques in the past 30 years have really made headway in breaking this log jam. Deep nets have just tilted the playing field in its favor, so long as we're willing to ignore fussy details like dependence, FOL, and causality.
The work by Tenenbaum at MIT is a particularly interesting mix of psychology, computational statistics, and neuroscience. It really challenges the commonly repeated notion that neural networks are completely unrelated to actual brains.
There is a direct trade-off between the expressiveness of your representation and your ability to reason about it.
If you include a semantics for supervaluationism (Sorites paradox, Keefe on vagueness)!
Thinking of it, I'd love to train a NN against human responses to '"is this a heap, yes/no?" +/-1 grain, repeat'.
The researcher, Prof. Levy, is French :)
Imagine a speech, by president Obama. Where you can actually hear his words. or imagine a ball dropping. We just use words to convey the basic idea. How many hairs does a bald person have?
And if we have a misunderstanding we just adjust our system. New word we use context, including social context etc.
I totally agree.
How much of our “logical reasoning” about the world is just really strong correlations?
Look, AlphaGo Zero beat the best chess programs by just precomputing MCTS. What does that say about finding the optimal solution to a problem?
It seems MCTS beats even AlphaBeta search, by far.
So when you talk about applying rules, you’re using a system that tried to reduce the system to a few rules and maybe recursively use them. That may work for some things. Certainly it seems to work amazingly for physics. But that’s the unreasonable effectiveness of mathematics.
Ultimately what if I suggested that “understanding something” with huge vectors or monte carlo search is actually far deeper understanding than a few rules?
I am talking about the actual territory not the map. Yes the model is neat and tidy but the model is just simple enough so that humans can understand!
What if an AI can no more explain its solutions to a human than a human can explain human decisions to a cat?
No, probability alone simply isn't expressive enough to support higher cognition. At best it will deliver a purely reactive agent, no more cognizant than Kahnemann's knee-jerk level of thinking.
Also time is a local phenemenon. You can’t simply compare A and B time if A and B are really far apart. When you get to really small scales you would be hard pressed to find which event happened before the other, so you get uncaused events like Virtual Particles in quantum mechanics.
And when you have tons of complexity then what do you mean by cause and effect instead of correlation?
https://www.quantamagazine.org/omnigenic-model-suggests-that...
BTW, precision isn't necessary for relative time to have great value in a huge fraction of possible world events. Obviously all you need is relative temporal sequence to disprove causality - since the batter can't hit until the pitcher throws.
In a great many event pairs, a lot of temporal precision Is unnecessary. Often it's only approximate relative time that matters, since the mental task intuits all the temporal constraints needed for understanding event coupling, like knowing the parent child relation implies the relative ages of each and much more latent info that may or not be relevant to a given interpretive task at hand.
So what you actually use is not Propositional Logic / Boolean Algebra but Bayes’ Theorem. You only use rules of inference because you make assumptions.
I am saying that may work sometimes but those models are not exact.
But from another side, current AI has a hard time to perform ordinary human reasoning at all. Those reasoning mostly contains a few logical clauses. Why?
Human like logic usually hide the common sense or context in other words. There is no absolute logic like FOL when we talk about human like logic. The traditional rule based AI system fails because of that.
On the other hand, deep learning are very strong at representing these hidden common sense and context. Look at the word2vec.
We now are at a point to combine these two approaches together not to build AlphaGo, but enable ordinary human like reasoning.
At least that is my hope.
It doesn't mean "deep and meaningful".
It would not be nearly as well funded were it not for this misconception on the part of budget holders, and everyone involved in ML knows it.
The previous name for what is now called “machine learning” was predictive statistics. But a better name for deep learning is “machine intuition” since it seems to work pretty well but can’t be readily explained
It works "pretty well" in well-defined complete-information domains like Go.
The training in that case is actually about improving the next dataset from which higher-quality patterns are extracted.
1.We have a NN that doesn't know how to play and makes random moves.
2.Blank slate dataset filled from random games.
3.NN learns to produce better games by weighting wins vs losses, making losing moves less likely and winning moves more likely.
4.NN produces slightly better dataset.
5.Repeat #3 until you have covered a large chunk of go openings and have pretty good estimation for move quality.
6.NN becomes the condensed statistic pattern data extracted from billions of games, able to estimate value of a move more accurately than humans.
Still in some rare cases it "condensed pattern database" will make mistakes due gaps in data or wrong correlation that NN hallucinated from its connections.
But no one can definitively say “my AI credit scoring system didn’t decline his mortgage because he is black” and that’s the explainability bar that needs to be passed.
Right now the state of it is “gut feel” with an unknown amount of unconscious bias.
That is becoming less and less true.
I always wondered if rules and facts can be seen as high-order abstractions that represent the low-level data that is encoded into neural nets. Would that mean that a system of symbolic logic, or rules and facts can in fact accomplish the same thing, just in a different, higher order way?
Then when you get into combining fuzzy with known, it's like different organs in the brain combining to serve different functions. When I know a rule and recall it, I believe it to be true and act immediately on it. When I'm not sure, I get fuzzy and try to deduce it. The more data and inputs I have about a given environmental condition, problem, subject, input, fact, etc., the easier it becomes to "reason" using fuzzy links to try to work out the answer. Unless I get information overload or the complexity exceeds one human brain's capability.
I've always felt that techniques are complementary and probably we employ multiple subsystems in our brains together to act like humans.
Perhaps the answer is to figure out all the things that make people tick via deep learning and other techniques, build one huge open api/db for vision/audio/movement in 3d planes/ethics/morality/understanding/discernment/language and then a more general ai could simply send out calls to api's of data already trained on specific foundations and create a 'whole' brain from multiple 'sub' brains (pre-learned open knowledge).
Unless we literally learn to clone the human brain technologically, in it's entirety.
I think another issue is not all working in AI are neuroscientists, and it may take understanding every mapped out facet of the human brain to come close to re-creating that in humans, it all can't be deep learning unfortunately, because that's not how we learn and we're the best guess solution to 'intelligence' as the only intelligent race (we know of) in the universe.
If you could humor me, why can't it all be deep learning? It seems like a robust model for how we learn: we start with some priors (random weights), experience something that either confirms or negates those priors (sampling), and then adjust our priors based on that experience.
Of course there are a wide variety of architectures and the human brain is more sophisticated than one conv net. Our brains have on the order of 100 billion neurons and I believe the largest neural networks are on the order of 10's of millions. I would argue that a combination of better architectures and scale could simulate human intelligence.
i dont think we start with random priors. We start with a set of priors that are geared for certain things - facial recognition, language etc. The environment then fine tunes them.
No, we don't "learn" how to balance, how to identify objects and navigate 3d environments. All of that is highly codified in our DNA from hundreds of millions of years of evolution. Human babies, similar to kangaroos, are just born too early for those systems to have developed.
Also, not everything we do is in our brain -- there's also muscle memory, and even tissue inside the gut stores some memories and 'knowledge'. Technically every cell in our body has some purpose/goal and can contribute to our entire being/feeling/mood/etc.
There's a lot we don't know about conscience that makes developing an artificial one, very tough. That's why it's not estimated to happen till after 2050. I don't doubt it will happen, I don't doubt deep learning as is will be crucial to the discovery, I just think there's possibilities that undiscovered and better/alternate techniques might be discovered that could potentially trump dl as we now know it.
There's a lot of things we do not know to get us to general ai, and to get there we may need to jump outside some 'boxes' that ai researches have put themselves into.
Other than being unsubstantiated, the real world only runs at 1x, simulated world can run at any rate your computing platform supports.
That’s the steelmanned argument at least.
The storing of that can take up precious resources, to create service droids that can do everything using AGI we need the computer that runs it to be essentially the size of a real human brain. Which we need to find ways to map AI tech more closely to biology.
There's also the fact that we are born with instincts built up over millenia, passed down. We also absorb random tidbits of data with no actual purpose. We watch the news and our toddler gets excited at the commercials, I don't know what he's learning from it, but getting him to turn his attention away from it is nigh impossible.
You can see the processing going on, it's amazing to watch him learn. I just think we need to think outside the current ai 'boxes' with dl/ml and consider their may be whole swath's of things we're missing, whole pieces of the puzzle before we get to AGI or the Singularity.
https://blogs.scientificamerican.com/guest-blog/a-brief-guid...
It seems there's nothing logical about human cognition, it just ends up resembling logic because movement through space either works or it doesn't, and humans have the ability to assign physical analogies to abstractions and to respond to those metaphors with the same heuristics and physical reflexes that it was born with; heuristics and reflexes designed over a few billion years to allow your ancestors to survive and reproduce.
The problem with training things on artificial datasets is that they are not rich (resolution is too low) or surprising (generated by simple algorithms.) An AI that worked as well as the human brain would just end up reading them back to you. The real world is, information-wise, incompressible. It has such fine distinctions that its impossible to examine them beyond a certain level without the act of observation changing them.
I always thought there was some potential in using robotic bodies with good sensors that fed into some processing theory of mind in order to grow an AI.
Simply put we need to work more on mirroring nature and biological brains to get computer brains that function like human brains. Even neuroscience still debates on what consciousness even is.
The answer I think, for the applications in industry at least, is that PRISM and other probabilistic programming langauges are not as developed as Prolog, a language that has been around for a good four decades now. Swi-Prolog in particular, is a free and open-source Prolog interpreter with an IDE, a graphical package and a veritable somrgasbord of libraries, including an http server, json and xml processing, encryption, interfaces to SQL and other databases etc etc etc. Swi is in active development, compatible with YAP-Prolog (currently the fastest implementation) very well supported and documented and has an active community.
It's a lot easier to setup a production environment in Prolog, especially using Swi, than in pretty much any other logic programming language, including the probabilistic logic programming ones.
Btw, I noticed that Kyndi list experience with Logtalk as a "plus" for one of their career opportunities (of course I looked :) which means that they're probably using Swi-Prolog.
Until the startup in question can make a compelling case for using prolog, much less avoiding the brittleness inherent in expert systems' resolution model, count me a disbeliever that their 'better way' really is.
IMHO, the choice of prolog unnecessarily straitjackets a modern production system, making the engineering approach of this startup that much less powerful, flexible, or viable than it should be.
Also, the (!/0) operator allows you to prune choice-points and control backtracking so that it is not unconditional. Many Prolog implementations support (->/2) operator, as well as ((*->)/2) -- the soft cut, for more fine-grained control. SWI has a nice library for even further control over backtracking as well: http://www.swi-prolog.org/pldoc/man?section=solutionsequence....
Also good Prolog programmers usually are not overly concerned with backtracking, as they usually have strong knowledge of modes and determinism. See: http://www.swi-prolog.org/pldoc/man?section=modes. This is mostly second-nature to a skilled Prolog developer.
But it is one thing to know that the current path isn't leading us there and another thing entirely to know which path will.
After all it isn't even clear that emulating biological intelligence would be a wise approach - we didn't build the jumbo jet by copying the birds.
(YMMV)
I was very pleasantly surprised this year to see the Symbolic Machine Learning course at Imperial College London well-attended by about 30 students each session. The course basically teaches the principles and practice of Inductive Logic Programming, a set of algorithms and techniques that learn logic programs from relational data [1]. It is an optional course at Master's level so I was originally expecting to see maybe half a dozen people attending (I assisted with a couple of the sessions and followed the rest to brush up on my background).
I discussed my surprise with my supervisor who is one of the tutors on the course. His opinion was that logic-based AI is much closer to the material that most Computer Science students are familiar with, than statistical and probabilistic AI. It's true that, although many CS courses have started to include statistics and probabilities in their curriculum (the one at Imperial sure does) the mainstay of computer science are logic-based, discrete maths, complexity and computation theory and language theory. The jump from there to GOFAI is just a short hop; in many ways, the two fields are really one subject.
My intuition is that the boom in deep learning is partially being fed by disciples of fields well outside CS -maths, physics, bio-sciences etc- jumping on to the, well, bandwagon, motivated by the deep learning frenzy in industry and not so much by any interest in AI or even computers. At the same time, there is plenty of traditional CS talent waiting in the wings. It may be that a revisiting of the good old ways of knowledge representation and inference may not be completely out of the question [2].
Of course, there's always the chance that the students only came to the course because it had "machine learning" in its name :)
__________________
[1] More the point, both of the outputs and inputs of ILP are logic programs, i.e. binary relations. Most of the work uses Prolog but there also exist techniques to learn Answer Set Progarmming, Constraint Logic Progarmming and of course Probabilistic Logic Programming programs. In any case, the homoiconicity of logic programming languages allows a newly learned hypothesis to be immediately reused as data to learn a new concept, in a way that is probably impossible with statistical techniques. And of course, learning is extremly sample efficient and the resulting programs generalise vey well.
I bet you had never heard of any of this before now :)
[2] To be honest, my guess is that the way forward is a combination of the two approaches- perhaps, some monster combo of deep learning for sensory perception, logic for inference and probabilities for final decision making.
One way or another, I expect the AI researcher of the future will have to be a jack of all trades - and master of all of them at once. Learn to juggle.
The human brain is unimaginably complex. Even if we created a neural network on the order of the 100 trillion or so connections in the central nervous system we would still not even be close. Each one of those synapse is a complex, constantly adapting chemical computer which is sensitive to the interaction of dozens of neurotransmitters. Gradient descent is to neural adaptation as minecraft is to photorealism.
The idea that artificial intelligence will be achieved by adding a few layers to a "deep" neural network, or by throwing more data at it is laughable.
It's called "cargo cult science" and it's an important part of the AI hype train to keep funding coming.
I have another reason to hate this. If we could give humans lidar to make their driving even better why shouldn't we?
Why should we deny an AI extra senses that can help it do its job better than we can?
Granted. On the other hand, electric signals can propagate across a chip several orders of magnitude faster than the chemical-electrical signals neurons use to communicate. So while our current path won't lead us to an artificial brain that works exactly like the human brain, it might still make one that's just as smart through pure brute force.
A biologist in 1902 could very reasonably have scoffed at the heavier-than-air flight enthusiasts who pointed at birds as a proof of concept. The biologist could point out that the complex, many-part motion of a bird's flapping wing was far beyond the crude mechanical processes inventors were trying to use to replicate it. And they'd be right. But in a much more important way, they'd be wrong.
I have no idea whether we're on the path to AGI or not, but I do think that "we're not building machines that work exactly like the brain does" is a poor counterargument.
It's not that artificial intelligence has to work exactly like human intelligence; my point is that the system that produces human intelligence is many orders of magnitude more complex than what passes for AI these days.
I think there's an idea out there that because deep learning allows computers to solve some problems that we thought were in the exclusive domain of humanity even a few years ago, that the refinement of these technologies and Moore's law will steadily close the gap between AI and human intelligence. I would venture to guess there are a lot of hard, novel problems left to solve before we get anywhere close.
edit:
It's also not the first time we've thought we had intelligence "figured out". In the middle of last century, there was a lot of hubris around the field of neuroscience as we started to understand the brain's topology, and could map specific functions to brain regions. The feeling at the time was that we would refine our map, and sooner or later once we figured out what all the parts did, we would have a total understanding of the human mind.
This paradigm led to some horrifically misguided practices. For instance the use of lobotomy to treat psychiatric disorders: if the brain is just a collection of functional regions, why not just cut out the bad bits right? Turns out it's not that simple.
You don't need to make a vehicle as complex as a cheetah for it to move as quickly as cheetah. You just strap a rocket to a box and you're there.
Granted a rough solution like that is not as good or as agile as a live cheetah in a lot of ways, but it depends on what you're optimizing for.
But if they're functionality equivalent, I'm cool with that.
Heck, it might turn out that humans are already just pattern recognition machines and nothing but. I'm cool with that too.
That was almost 10 years ago though and I haven't heard much of it since.