How Clara Labs (YC S14) Is Using Humans to Build AI | Hacker News Reader

How Clara Labs (YC S14) Is Using Humans to Build AI

blog.claralabs.com

90 pointsmarannelson30 comments

This is SO interesting! I am surprised by how companies are trying to solve AI problems... Machines normally take you 80-90% there, and using humans for that 10-20% left is an amazing idea. It's also a great monetization model for many companies.!!! Also, Clara is awesome :)

We think it's a good idea too ;) Machines and humans truly have different talents: machines are great at memory, keeping track of state, distributing information while people are great at understanding subtle nuance in natural language.

And in another 3-5 years, AI will start understanding subtle nuances too (probably) judging by the number of papers in computer language understanding.

Even modern methods today (deep recurrent networks, etc.) can do pretty well with these kinds of tasks (a very large ontology for instance) if you have enough annotations of the nuance!

>> Even modern methods today (deep recurrent networks, etc.) can do pretty well with these kinds of tasks (a very large ontology for instance) if you have enough annotations of the nuance!

Where do you get that from? NLP, with neural networks or not, stays as safely away from meaning as is humanely possible while working in an area very closely connected to it.

Also, ontologies? Very few people are interested in that nowadays, though that includes the team that made Watson. The push is instead to do away with all that and rely on statistical approximation.

Unfortunately "machines" (AI software, really) are really bad at nuance, subtle or blatant.

The number of papers in the field is a very good example of why: until you've read (enough of) them you have no idea what the state of the field is.

Edit: By this I mean that your assessment about the 3-5 years to "subtle nuance" is extremely, unrealistically optimistic. We're nowhere near AI understanding language. Try 300 to 500 years and you might be closer.

Not only that, but there are a lot of domains where you don't need to be terribly accurate in the first place, because existing methods are so terrible. A great example I encountered recently is identifying people who may be interested in switching jobs soon. How much do you think recruiters and companies would pay to increase their pool of interested candidates 5-10%?

To me this seems like the right approach - either you start with some massive amount of data that's not quite adapted to the problem (think Google), you start with full automation and have to basically write the decision tree yourself, or you generate the appropriate labeled training data like this.

However, it seems like there are some scale issues if you start upmarket like Clara Labs has been. I wonder if there's benefit in having a cheaper more mass-market version as well that can be used to generate larger amounts of data and test algorithms better?

It's certainly possible. One advantage of our setup is that rather than getting ok -to- noisy labels from customers, our CRAs understand the end-goal of the application and generate pretty great data. We are also able to incentivize them to produce fewer errors.

We've posted a follow up on the humans behind Clara here: https://news.ycombinator.com/item?id=12074657

Hi! I'm the author of the post and happy to answer any specific questions here.

How much human tagging time does it take to process 1000 meeting requests or more general how does the annotation side work?

Do you have people overlap items since there can be a fatigue issue as you mentioned in the post?

Clara has a 1-hr SLA for the processing of an incoming message. While I cannot give numbers on the speed of annotators (or volume), I can say that our platform is designed to enable quick and accurate work via incentive mechanisms. We avoid fatigue in part by making it easy for CRAs to navigate and work with data. Wrt to overlapping annotator schemes, these are known to be effective. We'll be writing more about how our human backend works in future posts.

Thanks, in the future post would also be interested in the trade offs between building internal system vs. using a 3rd party (maybe you do?) like mechanical turk and the incentive structure.

This is noted, thanks!

Our follow up on the humans behind is here: https://news.ycombinator.com/item?id=12074657

This is AWESOME. Human+Machine is always going to be better and stronger than machine alone. Keep it up! I wonder how this can impact the amount of people that can participate in crowdsourced projects.

Would love to integrate the ML and context aspects of Clara into our existing assistant team. Curious if that's something that's doable, or if it's just a package deal of their trained operators + AI.

But yeah, basically awesome and everything we'd need. Being able to go even more in-house and specialize our personal assistant and equip them with augmented capabilities here would be even more awesome.

Very similar to x.ai, although x.ai seems to be currently constrained to one problem domain (meeting scheduling). However, I'd bet they're looking to move be beyond just this.

https://x.ai/

Relevant: https://techcrunch.com/2016/06/07/crowdflower-series-d/

About how AI often needs a good UI to handle the last few percent of cases.

Navigating to

https://claralabs.com/

freezes my Linux system - a reboot is necessary. Is this a SPAM blog post?

Reminds me of the writeup a few days ago about Google's machine emotional intelligence for email suggested replys - except Clara is doing it (and more) with a team the fraction of the size.

The business solution here is very clever. As an engineer, my instinct is to fully automate any system. Clara is a great example where the most effective answer is getting the 90% solution, then going the last mile with humans to bring it up to 99% (making that up) accuracy, with most of the same savings. Imagine similar solutions for customer support (especially over website chat bars. Some systems like this already exist.)

Contemporary ML is really (and by that I mean human-level) good for some things: classifying images, guiding a car along prominently visible lane lines in perfect conditions, predicting timeseries that have latent patterns.

It's NOT good at the general "bot" problem (or should I say "true AI" problem), and some "features" of human assistants like EQ, generalized task completion, and complex planning are decades away from getting solved.

I would even argue that a mechanized/automated Clara would HAVE to pass the Turing test in order to work.

Your last comment on the seemingly circular relationship between Clara and a "turing-test-passing bot" is especially salient. This reference (found in the second footnote in the post) http://www.ijcai.org/Proceedings/95-1/Papers/125.pdf has a lot of interesting perspectives on the role of machine learning vs. human intelligence.

I am going to read that actually, thanks.

On the note of bots and your denouncement of the Turing Test as a metric, I do remember an article on here a couple of days ago hinting that one of the biggest frustrations of customer service calls is dealing with scripted responses, bots if you will, instead of real humans.

I do wonder whether alleviating this user frustration will amount to passing some "similar" but possibly weaker test.

"The customer service test" -- I like it.

Side note: your website's Javascript is broken. "i" is undefined.

Reply on news.ycombinator.com