Hacker News Reader

Comment by daveytea | Hacker News Reader

This is really cool. If you're looking for more datasets to train your model, here are a few relevant ones: - https://archive.org/details/stackexchange - http://trec.nist.gov/data/qamain.html - http://opus.lingfil.uu.se/OpenSubtitles2016.php - http://corpus.byu.edu/full-text/wikipedia.asp OR https://en.wikipedia.org/wiki/Wikipedia:Database_download#En... - http://opus.lingfil.uu.se/

I'd love to see how good your model gets.

View on HN

Thanks for the links! I'll have to try out some of these. The data is definitely the limiting factor at this point.

Have you consider project Gutenberg?

Is there any reason why that huge corpus would not be useful?

https://www.gutenberg.org/

I think the issue is you need parallel corpuses of (sentences with grammatically errors) -> (same sentence but correct). Gutenberg has a lot of only the latter I think.

You don't need parallel corpora -- the OP was generating the incorrect versions of training data through random perturbation (dropping articles, etc.)

The author can probably get a lot of traction by doing random perturbation, but they won't be getting things in the correct ratios (i.e. lots of dropped hyphens but not many dropped apostrophes) and also won't probably make all the types of errors that humans actually make. It will work, but a huge part of machine learning and doing NLP is getting those ratios right. This is one reason Google slays with their translator, they have huge corpuses that allow them to get fine grained distinctions in their models.

Reply on news.ycombinator.com