Hacker News Reader

Comment by andrea_s | Hacker News Reader

For all the hype that surrounds this platform, the live demos are truly underwhelming...

Agreed. For kicks I dumped a bunch of dmr biographical material (Wired, NYT, his personal web page) into http://ibmlaser.mybluemix.net/siredemo.html

Some "PERSONS" it found, everything between '' is [sic]:

'C' 'Cobol' 'B begat C.' '”' (yes, a single fancy closing quote) 'engineer' '’ initials' '’70s' '“Jobs’ genius'

Some of it is understandable given how annoying single-character identifiers are to parse and others are roughly reasonable in context.

Even given some of those quirks I bet it would be useful if you had bajillions of pages (cough ediscovery cough) to search through instead of a few general-audience articles.

For anyone else who wants to know, 'dmr' in parent post is Dennis Ritchie.

(I knew this, but didn't make the association right away, and it's quite hard to search online for 'dmr')

Searching for 'dmr person' does the trick with Google. Top result, and a sidebar of Dennis Ritchie.

You should've asked Watson. Then you would have learned the truth: dmr is 's wear daily.

Yeah it's pretty bad, even inputing a well written news article like this one: http://www.theguardian.com/fashion/2014/oct/08/return-of-joh...

    ”              PERSON
    its            ORGANIZATION
    ’s Wear Daily  PERSON
    shoulder pad   DISEASE

The rest is mostly correct (although random punctuation seems included at times) but it's nothing you couldn't get by word matching using a big dictionary. It reminded me to get tested for shoulder pads though, so there's that.

We have a bug with quotes/punctuation that we are fixing now (that account for 1 & 3). "its" is not necessarily a mistake, it likely refers to the owner. "shoulder pad" is a mistake due to the probabilistic nature of the algorithms used.

FWIW I ran the article through AYLIEN's Text API and here's what I got: http://aylien.com/text-api-demo?text=http%3A%2F%2Fwww.thegua...

(disclaimer: I'm the founder of AYLIEN)

We have found a bug with character normalization (which messes up the parsing of the punctuation). We'll fix this today.

Interesting, this service is a lot like http://www.alchemyapi.com/products/demo/alchemylanguage

We fixed the bug with punctuation. Can you try again?

Took me a while but I retried. It's definitely better, to the point that I have to stretch a bit to find really bad errors. "B begat C" still makes it in as do a few similar ones but considering how pathological a case a single-letter identifier is I can forgive that.

I assume because the identifiers aren't unique that they're returned in the underlying api as text locations. Hmm now I want to throw EDGAR data at this.

I played with language recognition demo and learned that Polish is apparently German. Being born in Galicia[0], I guess I shouldn't be surprised...

[0] - http://en.wikipedia.org/wiki/Galicia_%28Eastern_Europe%29

Sorry Polish is not one of the supported language yet: Arabic; Chinese (Simplified); Chinese (Traditional); Cyrillic; Danish; Dutch; English; Farsi; Finnish; French; German; Greek; Hebrew; Hindi; Icelandic; Italian; Japanese; Korean; Norwegian (Bokmal); Norwegian (Nynorsk); Portuguese; Spanish; Swedish; Turkish; Urdu.

Oh right, I'm sorry. It was actually written on the demo page, but I somehow missed it.

I also imagine that distinguishing between many of these languages is not a trivial task for a computer to handle.

You'd get collisions if you're trying to determine the language of a single word, obviously. But a sentence or two should be no problem. You're just matching words in a dictionary.

Different languages are mostly quite distinct. It would be a more interesting challenge to recognize and understand, say, German dialects.

In fact you can already get very good results, even without a dictionary, just with a distribution of the trigrams in your input languages (which is immediate to compute from a corpus). Just compute the distribution of the sample and return the language that has the closest one.

Even a single letter frequency histogram is already a very powerful discriminator to detect language. Although I suppose scoring maximum likelihood on bi- or trigrams works better for very small (sub-sentence) bits of text.

Odd that Farsi is supported, not Polish, given that Poland's GDP is close to Iran's and that Poland has no sanctions against it. Perhaps IBM is planning to sell lots of Watson-powered systems in Afghanistan & Tajikistan?

Incidentally, Poland's GDP is several times that of Finland…

Polish as a language is more similar to other languages than Finnish or Farsi. Basically, it's easier to detect Farsi and Finnish because they're more distinct and in smaller language families than Polish, which is similar to many slavic languages.

Do you not think there might be a market for software for analyzing text in Farsi perhaps -outside- the region where it's actually spoken?

(disclaimer - work for IBM, but not on Watson, or anything related to intelligence analytics)

French is identified as EN_us -> http://imgur.com/VdDJVw4 Inadmissible !

AYLIEN's Text API does a better job :-) http://aylien.com/text-api-demo?text=vache%20cochon%20metal%...

(disclaimer: I'm the founder of AYLIEN)

Do you mind sharing your input text? Interested to see how our API performs (http://aylien.com/text-api-demo)

Reply on news.ycombinator.com