Hacker News Reader

Comment by liliumregale | Hacker News Reader

The paper has recently been called into question for overestimating their performance relative to BERT: https://news.ycombinator.com/item?id=36758433. Might be good for the blog's author to take this into account in their explainer. The author's perspective sounds a bit too positive (and borderline salesmanlike).

View on HN

The second to last section "some potential issues with the paper" discusses the top-2 finding.

Yes - it's mentioned, but doesn't the framing below make it sound like they're still advocating for this paper?

> In essence, it's advisable to take the paper’s reported figures with a grain of salt, particularly as they cannot be precisely reproduced as described. Nonetheless, this approach continues to deliver unexpectedly well.

A "grain of salt" is different from "critical evaluation flaw," and if the reproduction's results are true, then the method doesn't after all "deliver unexpectedly well".

I take your point that it could have been more strongly worded. The reason I say it "devliers unexpectedly well" is because the whole concept of using gzip for classification is unintuitive, and even after fixing the flaw it still manages to get decent accuracy (given that it is no more beating state-of-the-art models).

Further analysis shows that it doesn’t perform well at all—successes are tied to things like test set leakage.

https://kenschutte.com/gzip-knn-paper2/

This paper isn’t any surprisingly effective result. It’s thoroughly shoddy scholarship by which the authors should feel embarrassed.

Thank you for reading :-)

I mentioned it towards the end, in the 2nd last paragraph. Those issues in the evaluation do bring its accuracy down a bit, even then it performs better than expected, considering it is doing knn on compressed data.

Well…one that peeks at the test set labels.

https://kenschutte.com/gzip-knn-paper2/

Reply on news.ycombinator.com