Hacker News Reader

Comment by nl | Hacker News Reader

I'd be lying if I said I don't doubt you are not incorrect ;)

What you are proposing is a statistically generated version of the Gunning Fog Index (http://en.wikipedia.org/wiki/Gunning_fog_index) or the Flesch–Kincaid test (http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readabil...).

If I were Google, I'd try that, but I'd also try something like working out percentage deviation from a Markov chain generated from their crawl. A method like that would show that my first sentence is pretty unreadable, while an algorithm based on word complexity would see it as pretty simple.

View on HN

> I'd be lying if I said I don't doubt you are not incorrect

Holy Shnikes, that's a tough one to parse! I wasn't sure what you were saying here, so I'm gonna break it down, working from the end of the sentence:

1. I'd be lying if I said I don't doubt you are not incorrect

2. I'd be lying if I said I don't doubt you are CORRECT

3. I'd be lying if I said I don't think you are INCORRECT

4. I'd be lying if I said I think you are CORRECT

5. I think you are INCORRECT

The idea is that each of the previous statements are saying basically the same thing; I'm just cancelling negatives each time. Anyway, am I correct to assume that you think that the GP is incorrect?

Your transition 2 -> 3 is not justifiable, "doubt" is not the opposite of "think". A better parsing leaves "doubt" alone and would end with: "I doubt you are correct."

My dear, you just lost the "don't".

I'd be lying if I said I don't doubt you are not incorrect

I'd be lying if I said I don't doubt you are correct

I'd be lying if I said I think you are correct

I had to triple check to make sure I got it right, and I wrote it.

Yes, I think the GP is incorrect.

> If I were Google, I'd try that, but I'd also try something like working out percentage deviation from a Markov chain generated from their crawl.

Indeed, that was my second thought, but I wonder if the gains are really all that large over a raw statistical analysis of the word bag, and whether they're worth the extra analysis space/time. It really depends on what Google is planning on doing with this metadata, internally; if an order-of-ten precision is fine (to pick out decisive categorizations), the raw analysis may be all that's needed.

Well we could always try it out. Here's 24GB (compressed) of ngram data from Google: http://googleresearch.blogspot.com/2006/08/all-our-n-gram-ar...

Reply on news.ycombinator.com