Natural Language for Developers | Hacker News Reader

Natural Language for Developers

wit.ai

183 pointsPerados56 comments

In it's current implementation, Wit.ai is broken.

I built a bot recently on it and it was one of the worst experiences I've ever had integrating into a third party API.

By a long shot.

1) chat logic is declared on their buggy web platform, which is inconsistent if you use multiple tabs, and doesn't actually indicate current system status. I had a number of "bugs" whereby my "story" was correct, but attempts to test it failed and only after I gave up and let it be for a few hours did it start to work. Meaning stories that were broken would all of a sudden be marked green after a few hours. Flows that failed would start to work given some unclear quantity of time. Oh yeah, and the opposite too: things that worked while futzing around with their interface would potentially break after a few hours. Delightful.

2) Documentation is horrific. It's not clear from any of their examples how to build anything other than a toy to turn on and off lights. If you can figure out how to build a "yes-or-no" entity that follows two different paths from their "recipes" I will buy you a drink ($10 USD and for the first five people). Just provide proof and a BTC address.

3) Flow control is next to impossible to implement, and when you do, it's totally opaque as to why the bot actually chose one flow over the other.

4) can't currently version your stories, which means you can develop new features without breaking production.

6) their support is horrendous. I have four outstanding questions, all unanswered after, oh, two weeks?

The combination of all of the above means it's INCREDIBLY brittle, hard to reason about and extremely time consuming to develop against.

If Facebook wants people to build bots for their messenger platform, open source wit.ai, and then pay people to develop it. This current state of affairs is untenable, morally questionable for all of the reasons outlined elsewhere in the thread (which have gone unsatisfactorily answered by apparent Facebook employees who should know), and fundamentally broken.

So, the site has this demo: https://wit.ai/getting-started First two queries it shows are "Turn on the lights" and "Open the door please!" showing how nice the response is. If you combine them in "Turn on the lights and open the door please!" then the reponse ignores the door part completely: { "_text": "Turn on the lights and open the door please!", "confidence": 0.94, "intent": "lights", "entities": { "on_off": [ { "value": "on" } ] } }

"wash the car" , "feed the cat" results in

  "intent": "tv_onoff",

seems wrong for such simple example.

Sounds like the algorithm was trained by someone with heavy procrastination problems.

Similarly, when I input "make me a sandwich", I got "tv_onoff" as the intent.

I am assuming however, that you can "train" the AI to suit your particular syntax depending on your app and bot's intended purposes. No doubt as more scenarios are fed into Wit's database, the parsing and intent selection will get better?!?

I typed "Translate how much does that cost to French", which was one of the examples it showed me on the front page (under "Intent Parser", exactly as shown, no quotes), and it still gave me tv_onoff.

And "Turn on the TV please" is interpreted as "Turn the TV on to channel 'please'".

And don't invite any physicists round, because "Set the temperature to 293 K in my bedroom" results in a "temperature" value of 293000.

Still, the docs do indeed say you can train it (for sentiment and yes/no queries, at least, and hopefully other things too).

Who leaves the tv on when they wash the car and feed the cat?

It was awful. It washed the cat and fed the car.

could have been worse - it could have come out with "wash the cat", resulting in serious injury!

It seems like Facebook built this so they can feed the data from people using their api to train their eventual messenger AI/assistant.

They were actually acquired by Facebook some time ago.

I don't know why you are being downvoted, but this reminds me that thing about facebook/google/etc. that if there is no product you are the product. It also came to my mind that they may be after your users data for their own models..

They "may be"? As far as I heard, Google is pretty clear about using spoken language input for their voice recognition API for precisely this purpose.

Correct me if I am wrong, but this approach essentially recognizes entities and matches it to a rule. This is naive ! There are better approaches which work based on parse tree structure or use a PCFG to encode rules

Can you link to these better approaches?

PCFGs are about the first thing that I learned about in my Stastical Machine Translation course. It's like a regular context-free grammar (see Wikipedia if you're not familiar with these), but each production is assigned a probability, such that all productions from the same non-terminal sum up to probability 1. Then you can assign a probability to a parse tree by multiplying the probabilities of all productions that were used (the model assumes that all productions are mutually statistically independent).

After a proper training run, you have your ambiguous grammar which, given an input sentence, will produce a set of parse trees, and assign a probability to each of them. You will typically choose the most probable one.

The Wikipedia article on PCFG is surprisingly biased on RNA/protein analysis and does not go into much detail on how to train these PCFGs. But it's basically just an EM algorithm using inside-outside weights. See for example https://link.springer.com/chapter/10.1007/978-3-540-78291-9_...

This requires defining matching patterns, like a word-based regex though, right? Houndify uses that approach because it is easier to integrate with speech recognition, but it is very cumbersome to use and maintain and doesn't allow learning new utterances.

I don't think Wit.ai works as simply as you believe - it isn't merely matching keywords. Unfortunately I haven't been able to find out exactly how it does work. Alexa, and Nuance Mix work the same way as Wit.ai. In my experience it is the superior approach.

Viv is also one to watch but they haven't really said the NLU works yet.

I've been using Wit and enjoy it. The site could improve in terms of usability and performance. It definitely feels buggy at times.

I'm not using their story framework though. Just their intents parser. Using wit entirely for the conversational layer doesn't seem like a good way to go about things.

Overall I'm happy though.

- I wish I could download my data set at will.

We will be releasing an import/export in the next 2 weeks

Wonderful! Thanks!

I'm using wit.ai as well, and overall it's working well for me.

But, the new story-based functionality is something I still haven't been able to get comfortable with -- and it still feels very early. So, that part I'm still implementing myself.

And yes, I too wish they they would allow you to download your logs/model. Bugs me that they don't do that (and one of the reasons I'm also looking at api.ai).

Yeah I took a look at api.ai but the reason I ultimately went down the wit.ai path is because of Facebook backing them. Seems like a nice wallet + data set to help.

I was using wit.ai for adding an NLP interface on an IoT product at a startup a few years ago. This was before wit.ai was acquired by Facebook.

They have an impressive product and a great team. It was very smooth to work with their api, and with reasonable accuracy.

I've used nlp-compromise.js for building an ecommerce shopping assistant bot. It's a bit rough around the edges but works most of the time.

Demo: http://nlp-compromise.github.io/website/

Github: https://github.com/nlp-compromise/nlp_compromise

You seem to have commited a merge conflict in https://npmcdn.com/nlp-syllables@0.0.4/builds/nlp-syllables....

Library looks fantastic!

I have been using this with https://jasperproject.github.io/ on a raspberry pi to make an open source amazon echo type device. It works pretty well for controlling basic things around the house.

How does wit.ai make money?

Wit.ai has been around for a while now, and used to charge a monthly fee for some private models. It was then acquired by Facebook and made free for all uses.

I guess it would really improve peoples' trust in wit.ai (and thus the number of users) if they would declare their plans for monetization or policy otherwise for the future. Otherwise developers might not want to see their work become invalid when the platform suddenly costs money, or something.

It's owned by Facebook.

What part of their monetisation plan (privacy whoring to sell ads) is not apparent to anyone who cares to look by now?

That would be rather short-sighted. Different platforms should be monetized differently, and Facebook has shown that it is willing to do this, such as with WhatsApp, which isn't filled with ads: https://www.quora.com/How-does-WhatsApp-make-money-especiall...

With wit.ai, I would be surprised if they decided "privacy whoring to sell ads" is the right way to go.

From that very page

> Instead, the company said it will explore ways businesses can use WhatsApp to connect with individuals, and will introduce new ways for users to communicate with businesses and organisations that will pay the company to target relevant communications with customers.

>> organisations that will pay the company to target relevant communications with customers

So, their plan is more targeted advertising in the form of "offers" or "deals" etc.

Ah, I think you may have misunderstood. Below those lines, the examples given on the Quora link are:

> A bank could use WhatsApp paid account to communicate with its customers about recent transactions and necessary fraud warnings.

> An airline could use WhatsApp paid account to contact its passengers about a delayed schedule or cancelled flight.

I would argue that these do not seem like ads.

Right. Just like my telephone company says it might want to send me SMS notices about when a bill is due, and then actually sends me SMS notices about every fucking promotion under the sun from their sister companies, in a language I can't read.

I honestly do not understand the logic of people who look at a company, wholly owned by Facebook, and give them the benefit of the doubt when it comes to business tactics, privacy, etc.

Would never use that. It's free of charge, so how is it financed? Right, its Facebook owned and just like every product of (Google, Facebook, Apple, whatever) if if cannot pay for it I won't use it, they eventually stop offering the service someday because they change their plans. Don't want to be forced to rewrite my app because of their change of mind.

These are my fears too. I've been looking at this over the weekend as a possibility to embed a 'bot like feature into my web app, but I would hate my users to get used to it, then have Wit go away like Parse did. Much harder to develop an AI engine from scratch myself than an authentication/database engine.

Give Abot a look (https://github.com/itsabot/abot). It's an MIT-licensed bot framework, and it should do everything you need. Would love your feedback as well--email's in my profile.

Exactly, Parse was a great example. Rather basic service and the outcry was huge. NLP is way harder and maybe even impossible for most developers. But I'm optimistic, that something like wit.ai could be done in an opensource community-driven environment. Projects like SpaCy[0] make NLP easier to use for more developers.

[0] https://spacy.io/

As far as I've seen, this isn't anything you couldn't do with basic classification and NER passes in OpenNLP. Straightforward, even trivial, stuff as far as existing NLP toolsets go.

Wit seems to be heading towards more conversation-oriented (ie multi-message) approach to NLP which is something existing NLP toolsets don't make straightforward. For example, predicting the next action a bot should take based on conversation history is a bit different than just classifying an utterance. There are hints at how to do this in literature, and Init.ai (disclaimer: I work there) is working on it as well, but it's not widespread.

There's also more to this type of slot filling than just NER. Again, the necessary techniques are available in academic literature, but not necessarily turn key. Plus, you need to handle parsing after locating the slots. For the parsing, you can take a look at Duckling (https://duckling.wit.ai/) which Wit did open source.

Both on the classification front and the slot filling, open source toolkits might get you part of the way there but not all the way.

Having a training and management UI is also a substantial value add once you use it.

The voice recognition isn't basic. The Open Source equivalents are no where near as good, unfortunately.

We understand the concern here at Wit. It's free because we no longer need to generate revenue. Wit developers spend time validating data that is critical to improve the models for the Wit community. When we joined Facebook in Jan 2015, we updated our Terms of service to mention that you can retrieve your training data if Wit closes at some point. FYI, the training data (expressions, intents, entities) is the critical component. The plan is to continue supporting and growing the product. The release of Bot Engine Beta a few months ago is a proof.

The fact that you're saying "it's free because we no longer need to generate revenue" is exactly the problem. It means you didn't even think this through. If you had, your answer would be different.

It's just like Twitter saying "We don't need to generate revenue directly from our API and that's why we're opening it all up because what's good for the community is what's good for us!", except that this isn't 2006 and everyone knows better.

It's free because Facebook wants the training data. Pretty obvious I would have thought.

If you're ok with helping Facebook improve its AI then I see no reason to worry about the fact that it is free.

Our vision is to make it easy for developers to make apps or devices that can leverage AI/NLP/Speech. As you probably know, the key is the amount of data you can collect. The more data, the better your models will be, the better you API is. When you are a startup, you need at some point to generate revenue to continue to grow/hire even if you objective is to gather more data

I came to post the exact same comment. I would never lock me in in such a naive manner.

Core components such as AI, or even translation shouldn't be outsourced in my opinion, at the very least to be cost efficient, or as in the case of this product, to avoid dramas which can be foreseen.

Business wise and sustainability wise, reinventing the wheel is more often than not better than using a SaaS. Commodity hardware-as-a-service (servers, service bus etc...) is the only exception (but still is far from being a no brainer).

At the end of the day, after all the time and money we spent in college, we should seek a ROI by implementing academic papers whenever we need to.

> Would never use that. It's free of charge, so how is it financed? Right, its Facebook owned and just like every product of (Google, Facebook, Apple, whatever)

I understand, and agree with your basic sentiment and point. I don't get why you include Apple with Google and Facebook.

Google and Facebook famously make all their money from offering 'free' services and monetising eyeballs on ads.

Apple famously makes all it's money from (traditionally) selling hardware, and recently, through it's digital media stores.

Apple give very little away "for free" - sure, they don't charge for the OS, but its financed by the purchases of hardware, not being a privacy whore.

Included them because they tend to shut down free services. Google shuts down nearly every service companies they buy offer. They are known for doing so on their very own projects as well. Oh and Apple does this as well, for example when they bought Beats, they shut down the Beats API.

> Included them because they tend to shut down free services

What Free service has Apple shut down? Their major that's been around for a while is iCloud/etc, which before was MobileMe, which before was .Mac, which before was iTools.

I don't know the technical details of how those back-end services relate to each other (i.e. evolution from one to the next, being rebranded, or re-built from scratch each time, etc) but there are very few things that they've truly shut down.

For a while after MobileMe and before some of the major improvements to iCloud, Keychain sync was not available, but I'm struggling to think of free services they've actually shut down.

Foundation DB http://techcrunch.com/2015/03/24/apple-acquires-durable-data... "Apple has acquired FoundationDB, a company that specializes in speedy, durable NoSQL databases, TechCrunch has learned.

A notice on the FoundationDB site notes that it’s no longer offering downloads of its database software. "

Granted, Apple doesn't acquire at Google's or Facebook's rate, so the sample size is smaller

Not a service.

I have the same thoughts. Would love to use it for my business, but I just doesn't feel safe using it, when it can close with no notice at all.

It happened to Parse, but thankfully they open sourced most of it do you can use it as a stand-alone.

You could probably architect your app in a way where you can swap out wit for something else later on, if the need be, without breaking the clients

Technical practicality is probably less of a concern. If you build your business on the availability of cheap (or free) APIs, and then these APIs cease to exist and you are left with very expensive alternatives (or worse, no alternatives at all), your business plan may break down pretty quickly.

The same could happen if the API was paid. In this instance, it's conceivable to setup your own NLP/bot engine with OSS alternatives to do more or less the same. But starting with wit would be much faster and allow you to focus on getting the product market fit right

What we now need is a real AI engine.

Reply on news.ycombinator.com