Hacker News Reader

Comment by rspeer | Hacker News Reader

The baseline I'd like to see this compared to is the not-very-deep-learning "bag of tricks" that's conveniently implemented in fastText [1].

[1] https://github.com/facebookresearch/fastText

View on HN

Great point! I considering using fasttext as a baseline, however in practice fasttext really didn't work well at all with the small data set, much worse than the tfidf baseline. I think Fasttext's classification approach might not work well with such a small dataset. I'm not sure but I suspect its because it tries to learn embeddings - but there just isn't anywhere near enough data for that. I'd love an outside perspective on this.

Fair enough. That's a useful comparison to know about.

But I'm wondering how you get around that with the neural net. In the post, you said there are only a few hundred labeled examples, right? How can a neural net with hundreds of parameters set those parameters to anything reasonable, and not overfit, when there are about as many parameters as examples?

Great question and I share your intuition but I think its all properly regularizing your model. I guess for neural networks, Dropout works really darn well as a regularization strategy. I could have tried to see whether performance dropped significantly without dropout.

why not vowpal wabbit? fast text is practically a ad-hoc version of mathematically proven vowpal wabbit?

I've used both. fastText will openly tell you that it's basically Vowpal Wabbit but faster (hence the name), with a way to stick word embeddings in without pre-processing, and with fewer ways to shoot yourself in the foot.

Reply on news.ycombinator.com