Hacker News Reader

Comment by edblarney | Hacker News Reader

I've worked in computational linguistics:

1) It's 'hard' because you need a lot of 'training data' in order to train models etc.. It's hard to get.

2) 'AI' type interfaces represent a whole new kind of UI challenge. For 'predictive typing' for example, you can optimize an algorithm so that it does better for 90% of the US population, but then it gets 'worse' for the remaining 10%. So it's a paradox. This can have weird effects.

For example, if you have an app in the app-store, you may leave the settings so that it's 'broadly optimal'. You get ok stars.

If you then make it 'better' for those 90%, you might get a little boost in ratings, but you get 1 and 0 star ratings from the 10% for whom it's a sub-par experience. This can destroy your product.

Anyhow - 'there is no right answer' often in AI, and setting expectations can be extremely difficult.

And all of that has nothing even to do with CS.

View on HN

That reminds me of Stephen Hawking's computer. His old computer gave deterministic dictionary predictions and after some time, he had memorized all the predictions that came up, so he could be much quicker because he knew exactly how much of the word to type and then to select the proper word - a purely mechanical process that doesn't require feedback anymore. However, when they gave him a new computer, it had adaptive predictions (I think) and Hawkins could not memorize them anymore. Then he always had to type letter for letter and inspect the predictions before he could select one - a huge step down.

Stuff like that is absolutely crucial but is often forgotten by engineers.

Seconded, and yet it's madness to me that engineers don't pick up on this. I can't be the only one who's let someone use my computer for basic internet browsing and then been deeply irritated when their newly visited sites break my intuitive knowledge of how many characters I need to type in the URL bar before hitting enter...

"And all of that has nothing even to do with CS." -> therein lies the kicker! Consumer level (including sass) apps require very real user and business tradeoffs. That tradeoff is rarely discussed, which i honestly find very curious in and of itself.

That is about it. Only need to add the actual data quality that matters, too: I often get a ton of junk to work with, which is partially useless. And the difficulty isn't just "algebraic debugging", but embedding the whole pipeline in a way that won't blow up or grind to a halt the it's used in a production environment, especially during peak loads when "everybody is looking" or when a new semantic event type happens.

I appreciate that you've given a pretty topline overview but in the 90/10 example, if that 10% can be characterised and/or clustered can the algorithm be optimised for both groups involved? Appreciate that that's not always possible and can lead to lots of engineering overhead - curious what your thoughts are though...

It's a very good point.

Yes, often it is possible to determine where the user belongs in that 90/10 setting, but it can take a lot of time in order to be 'pretty sure'. You need a lot of 'user interaction' in order to make that assessment.

The 90/10 rule can broadly apply to things like culture: certain Latino Americans speak/write very differently. A lot of 'le' and 'la' (gendered) in there as well as a whole different set of proper names and colloquialisms.

But it can take some time to really establish if someone is 'latino' from their writing.

Even harder: some people type more precisely, some people type more loosely. You can actually adjust the probability spectrum of a predictive keyboard to match someone's style. But get this: people's style changes all the time! I noticed that when I'm tired, I type like I'm drunk. Or if I'm busy etc.. So there's even variation in style that makes it difficult.

It's a really hard thing to do.

I should add:

You can 'massively decreasing returns to complexity' in these domains.

Meaning that you can do 'pretty good' with some basic algorithms.

For the next 'bump in performance' you need some complex code.

After that - you really start to have 10x larger models, or crazy complex engineering just to move the needle.

It creates a completely different set of 'Product Management' rules. It's kind of fun, unless you're a struggling startup trying to figure this out on the fly :)

Usually, someone comes along with a new approach which changes the games.

As I understand it 'Neural Networks' i.e. 'Deep Learning' style AI has changed everything voice related quite a lot.

And also - different business approaches can change the game. Google has access to zillions of phrases for properly transcribed audio phrases. This is the 'golden asset' that can underpin a really great voice recognition engine. Google voice is even better than the old industry standard - Nuance - in many scenarios and my hunch is that it's the size of their training data that has given them an edge - at least that.

>You can 'massively decreasing returns to complexity' in these domains.

This is a really concise expression I've been looking for the sentiments you've just laid out so thanks for that!

Really like your insight in Google, think it's spot on.

Re. 'Product Management' rules - would love to know more about this? Do you keep a blog?

Yeah, I was thinking the same thing. If you can tell which group a particular user belongs to then you can train two models and optimize them independently. Then you just select the most appropriate model for the user.

Practically this almost never works out. The 10% cluster is using a very small dataset and will produce inferior results. If you train a model based on only 10 people, you're prone to overfit that small sample

> 1) It's 'hard' because you need a lot of 'training data' in order to train models etc.. It's hard to get.

Also, I'd imagine that the data could be bad/incomplete, e.g. data was collected in an inconsistent manner or in the wrong areas, leading to an incorrect solution that fits the data, but doesn't solve the problem.

This is the biggest concern I have in using the data that we've collected to come up with a solution using ML: no one ever intended for the data to be used for the purpose for which I would use it, and is incomplete or incorrect.

However, I think the chance of good things coming from inadequate data outweighs not trying to make use of the data.

Yup.

You HN lads are smart, you're pretty quick to figure out all the 'next problems' that one would encounter.

Yes - getting the right training data can be surprisingly hard.

Did you know how hard it is to get a 'very official' large set of words for a given language? It's hard!

There is no entity that really decides what language is - so you have to kind of determine it from what people write. But that takes a lot of writing, and frankly, you're making assumptions all the time there.

France has a body that's 'in charge' of their language so to speak, and most Western nations have entities that are 'roughly' that. Beyond the West, Japan and China ... it's a gong show.

'Filipino' is barely a language - even though many millions of people speak it, it varies in dialect from village to village and they barely resemble each other.

I think that someone will eventually come up with a 'probabilistic' OS because in the real world, nothing is certain ... some things are just more likely than others!

An official set of words is only useful if your NLP task is restricted to items that themselves are restricted in their language use. Twitter and SMS data sets are interesting because they represent something closer to casual speech rather than formal writing.

The French Academy provides an official dictionary and language usage, but speakers hardly restrict themselves to its contents.

Filipino mostly refers to the Manila dialect of Tagalog, whereas Tagalog is a language with many dialects existing in the Philippines. There are lots of languages in the Philippines but as far as I know they aren't referred to as Filipino.

For a lot of NLP problems you will probably have to make your own data set. It can be a lot of work.

Reply on news.ycombinator.com