Pursuing the Next Level of Artificial Intelligence | Hacker News Reader

Pursuing the Next Level of Artificial Intelligence

nytimes.com

30 pointsrockstar930 comments

Open articleView on HN

Article content is pretty empty, other than saying she uses statistics. Any pointers to her papers?

"Article content is pretty empty, other than saying she uses statistics. Any pointers to her papers?"

http://ai.stanford.edu/~koller/papers.cgi

enjoy!

Thanks.

This is a dupe: http://news.ycombinator.com/item?id=180047

>After three decades of disappointments, artificial intelligence researchers are making progress.

Do you think AI progress is going to continue accelerating for a while?

Old-style AI solved a few of the "smart" things that people do, such as playing chess and proving (simple) mathematical theorems. I predict that new-style AI will solve a few of the "dumb" things that people do, such as keeping our balance and getting a visual sense of our 3d location and orientation.

The difficult part will continue to be those areas where we're not just doing combinatorics and we're not just doing statistics. The meaning of a sentence is formed both by the grammar, which is nearly combinatorial, and by the meanings and connotations of the words used, which are hardly combinatorial at all. I doubt that this sort of thing will just fall out from bayesian networks, or from any simple development based on bayesian networks.

Language will be difficult for quite some time due to its links with culture and the fact that it is always evolving and cross polinating between languages.

I don't believe there's any way to solve that without an AI that can actually learn and drive it's own understanding by observing and interacting with its environment both physically and through mimicry.

"Language will be difficult for quite some time due to its links with culture and the fact that it is always evolving and cross polinating between languages."

How much trouble do you have understanding things written in English 100 years ago? 400? 800? Older works require footnotes, but they don't introduce new verbal concepts -- you can footnote Chaucer in modern English, and I suspect that a Chaucer scholar would be able to describe modern concepts by building on old ideas -- there isn't a discontinuity where you have some concept that simply cannot be linked to the words Chaucer would have used.

So the problem is not that language changes too fast.

I think what he is saying is that the generation and extraction of concepts requires context that language may or may not provide. If you think about the approach of AI with language, it is almost the opposite of how humans learn: presentation of grammatics/language first and the extraction of concepts from this, whereas humans learn basic concepts of reality before speaking, and language is a partial representation of what they know.

That's true, but it's not at all what he said. He argued that language is a hard problem because it's linked with culture, and always changing. I think this is incorrect, because it doesn't stop humans from picking up second languages or from understanding older texts in their native languages. The hard part is whatever loop humans have that translates sentences into concepts -- not the loop that uses one sentence-concept chunk to link a new sentence to a new concept. That's in the library, not the interpreter.

That's true, but it's not at all what he said

What he implied from culture, I took to mean a polymorphic mapping of "sentences" or "units of communication" in a given understood context that can be learned only through experience.

it doesn't stop humans from picking up second languages or from understanding older texts in their native languages

It doesn't stop them, but it is far from trivial to learn a new language or to even resolve syntactic,lexical, or sematic ambiguity without context, even in a native speakers language: http://en.wikipedia.org/wiki/Ambiguity. The ill-posed nature of this mapping is not static, and for people not speaking their native language or young children this becomes the basis for some forms of comedy.

As far as I know we have no generic grammar that can generate the rules for all languages; there is no universal "interpreter", and it is not certain that "concept" can be decoupled from "phrase" or "sentence" without context, so I don't think your separation of "library" and "interpreter" holds.

That is in fact much closer to what I was thinking. The cultural context is important when it comes to understanding the less formal common grammar we humans tend to use more frequently and which is littered throughout the Internet.

We can understand 400, 800, etc year old text, but one of the first things you'll notice is the language has terminology and informal phrases that may die off, live on, or evolve over the years. Toss in ambiguity, metaphores, double entendres, etc and it's not as simple. Perfect example would be Shakespeare. His style is rife with creative linguistic twists.

Why would an AI need to interact with its environment in order to understand language? ISTM that deriving cultural information and understanding language change can both be found by simply observing its environment, and looking at the right corpus (e.g. the Internet).

>The meaning of a sentence is formed both by the grammar, which is nearly combinatorial, and by the meanings and connotations of the words used, which are hardly combinatorial at all. I doubt that this sort of thing will just fall out from bayesian networks, or from any simple development based on bayesian networks.

The most effective existing approaches to natural language understanding are purely statistics based. This works fairly well (about 85% error rate) in narrow domains. Most approaches aren't even bayesian network based, but use Markov chains/models which can be called a precursor to bayesian networks.

The most effective existing approaches to natural language understanding are purely statistics based.

This is where practicality and idealism ought to diverge. Sure, with enough data, we might be able to find spelling mistakes or figure out what pictures someone is looking for.

But the techniques we are looking for don't address understanding, (granted they never set out to). Most interesting would be to define meaning and understanding, using strict mathematical models. As McCarty said, language is just the froth on those waves.

Yes, we will keep making progress in AI.

There's two parts to the success of any machine learning program (like the stuff Daphne Koller is working on mentioned in the article). Your algorithm, and your data. To paraphrase Peter Norvig:

"Ten times more data will beat any algorithmic improvement."

http://glinden.blogspot.com/2006/10/advantages-of-big-data-a...

With the rise of the internet, more and more data will be online. Even if we can't figure out smarter ways of analyzing this data, we will at least keep getting more of it.

Here's a thought experiment. What if you had a really large dataset of chat logs, so large that any sequence of 1000 words, you could find it somewhere in the logs. Then you just wrote a chat bot that

1. Looked up the history of the chat in its logs

2. Said whatever the person in the logs said next

3. Once you got to 1000 words it said "Gotta go, talk to you later!"

Seems like that could pass the Turing test to me. Of course since there are about a million English words you would need on the order of 1000000 ^ 1000 = a googol^60 items in your data set. ;-)

>Do you think AI progress is going to continue accelerating for a while?

From where I stand, it looks like the more recent progress in AI is due to a shift in research hypotheses. It used to be like "AI is the attempted simulation, or sometimes attempted emulation, of human cognition"; now it's more like "AI is a medium for various forms of human expression". Ms. Koller's work emphasizes numerical expression; the work in Game AI emphasizes visual-artistic and/or logical expression, the work in Natural Language Processing emphasizes expression in natural language. Few people try for machines which pass the Turing Test; many try for machines which capture a small slice of real-life semantics and run with it. As Alan Perlis once wrote: "When we write programs that 'learn', it turns out we do and they don't."

I believe that we increasingly realize what he meant with this. And that's good, because not only does it say something about what we cannot do, but it also says what we can. People are increasingly aware of this, go pragmatic (Google/MapReduce proved that you can go quite a ways using brute force), and make progress.

I would think so, if for no other reason than Moore's law. As computing power becomes increasingly available, seemingly insignificant steps forward can become significant through the magnification of iteration.

is it true most AI was based on logic stuff? no wonder it failed. it's like making a building out of iron legos and wondering why it isn't flexible

For a good read on the subject of AI and how humans think check out the book OnIntelligence by Jeff Hawkins. It covers some about the previous attempts at AI and why they failed.

http://www.amazon.com/Intelligence-Jeff-Hawkins/dp/080507456...

That book is in my top 5 favorite books ever!

I keep hearing about this book. Can anybody tell me if it is a pseudo-scientific look at the topic? I don't mean this in a particularly negative way. Rather, what I mean is does it put forth a lot of as-yet unsubstantiated claims perhaps backed up with some hand-waving analysis?

It is not "scientific" in the sense that it does not prove its claims. But it's not "pseudo-scientific" either because it doesn't pretend it's proving claims. It's more like a book of interesting ideas from a smart guy written for a general audience. It's different from most AI in that it's very focused on drawing inspiration from neurobiology, while still being written from a "computer guy" perspective.

Here's an excerpt that might give you an idea of what it's like. It's in support of one of his key points, that many current AI methods ignore the role of change over time, whereas in your brain it's crucial.

"You know the alphabet. Try saying it backward. You can't because you don't usually experience it backward. Your memory of the alphabet is a sequence of patterns. It isn't something stored or recalled in an instant or in an arbitrary order. The same thing does for the days of the week, the months of the year, your phone number, and countless other things.

"Your memory for songs is a great example of temporal sequences in memory. Think of a tune you know. I like to use 'Somewhere over the Rainbow.' You cannot imagine the entire song at once, only in sequence. You can start at the beginning or with the chorus, and then you play through it, filling in the notes one after another. You can't recall the song backward, just as you can't recall it all at once. You were first exposed to 'Somewhere over the Rainbow' as it played through time, and you can only recall it in the same way you learned it."

Other interesting discussions are around the role of feedback and hierarchy in the brain's structure. I recommend this book to anyone who's a good programmer with some interest in AI.

Jeff Hawkins (the author, creator of PalmPilot) has always been interested in how the brain works, and once he made his Palm money, he started a neuroscience institute. That research along with his own and what he read led him to his theory of how the brain works. It's extracted from lots of real, hard research. He then started a company (Numenta http://www.numenta.com/) to create a software implementation of that theory, which has a set of tools you can download and play with. They've had good results consistent with the theories in the book and had some impressive results. There's also a recording of a talk he gave at an O'Reilly conference: http://itc.conversationsnetwork.org/shows/detail3499.html

So long story short, it is for real. When I read it, I had to stop every 20 minutes or so to catch my breath I found it so exciting and enlightening.

The book is great. His criticisms are spot-on.

His proposed solution is bold, but as of yet, unproven. It's probably wrong, and in a way, he admits to that in the book (he says we'll need a lot of changes).

If you get bored on chapters 6, skip it.

I second the recommendation of skipping chapter 6 - when I read this comment I went to check where my bookmark is because I had gotten bored while re-reading, and it was near the start of chapter 6 ;-)

Yes, when the field of AI first sprang up it was focused on logical deduction. In retrospect it may seem doomed to failure, but this is partially because our notion of "what artificial intelligence means" changes over time. There's a joke about this -

"Artificial intelligence is just whatever computers can't do yet."

Before chess-playing computers, a lot of people thought that chess required a lot of real intelligence, and if you could play chess well, that was a sign of intelligence. It's not unreasonable - for humans playing chess well is a pretty good sign of intelligence. But then we got chess playing computers that obviously weren't "intelligent like people are". So nowadays it seems obvious to us that playing chess isn't a real sign of intelligence.

So it's not that the logic-driven approach to AI failed. It succeeded at a lot of things that previously humans could do, but computers couldn't. Like playing chess. It just didn't get all the way to the Turing test.

  "My husband still berates me for not 
  having jumped on the Google bandwagon at the beginning,"
  she said. Still, she insists she does not regret her 
  decision to stay in academia. "I like the freedom to 
  explore the things I care about," she said.

That seems a bit silly. Given the computational infrastructure available at Google, it would seem that it's the only place where one would be free to try all kinds of ideas unavailable elsewhere. If she's as brilliant as they make her seem, I'm sure they'll let her work on whatever she wants.

did she invent bayesian spamfilters? i know there was ideas about it before paul graham did it.

http://www.paulgraham.com/better.html The footnotes here cite earlier work, as does the referenced Slashdot article.

Reply on news.ycombinator.com