Hacker News Reader

Do you expect your system to scale easily to the "the entire internet" data set? You mention that you degrade gracefully as the problem size increases; is getting the accuracy up to a usable figure only a question of more signals and processing power and dirty ML tricks, or do you think that you need a fundamental improvement?

View on HN

The algorithm achieves significantly higher accuracy if it has more text per author. Also, if you're willing to do human analysis on a few dozen candidates after algorithmically shortlisting them, that gives you a further advantage. Finally, there is much room for straightforward algorithmic improvement (e.g., ensembles of classifiers) that we didn't have time to fully investigate. In short, IMO it's just a matter of more data and slightly better ML, not fundamental improvements.

Pretty much what I figured. I look forward to seeing your future results! Or to seeing you to get shut down by the NSA. :P

Reply on news.ycombinator.com