> The system is only allowed to answer questions if it calculates that it is 97 percent or more confident in its answer.
Knowing when you don't know the answer is something that has been lacking from AI I've seen. Of course the problem is that not all AI has the option of deferring to a human.
Anyone know what percent of questions the AI answered, vs redirected?
That's actually a very good question. The article doesn't give many details other than: "There are many questions Jill can’t handle."
For comparison, here's are the highlights of a study from 2014 on the mobile assistants (Siri, Cortana, Google): http://searchengineland.com/google-now-beats-siri-cortana-di...
Google was able to offer "enhanced" answers (i.e. more than a web search) 58% of the time. Siri was at 29% and Cortana was at 20%. Siri improved some with iOS9, but not enough to beat google: http://www.recode.net/2015/9/20/11618696/how-intelligent-is-...
From the actual study: ... we took 3086 different queries and compared them across all three platforms. These were not random queries. In fact, they were picked because we felt they were likely to trigger a knowledge panel. ( https://www.stonetemple.com/great-knowledge-box-showdown/ )
I emailed and asked. If I get a reply I'll post it here.
Nothing wrong with that, of course.
Definitely, and I'm not sure why that space isn't getting more attention. For RL there's KWIK (knows what it knows) http://www.research.rutgers.edu/~lihong/pub/Li08Knows.pdf [pdf]. A large class of problems (the robo TA being one of them) lend themselves very well to AI with error bounds, for lack of a better term.