Comment by TZubiri | Hacker News Reader

Comment by TZubiri | Hacker News Reader

>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.

I disagree, but i agree that training with answers is worse.

In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing.

It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well.

It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases.

Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do.

And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other.

Hell, I'm not surprised that the software that the mass of test -studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes

> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question

Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs).

(There's also access to experts who know the answer, which kaggle bans by banning the internet)

Lucas describes it well here (and his original tweet up the thread): https://x.com/giffmana/status/2002128356901597509

--

Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2

I didn't phrase that well and cant edit, so clarifying:

What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"

>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

>What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"

This is getting into metaphorgotten territory, I'd have to do a deep dive to really understand whether it is "cheating" or, more formally, a methodology that encourages overfitting.

But for what is worth, taking the textbook to the exam (regardless of whether you were born there or not), would be cheating in a test as well. Although it is possible that in the model training context it does not lead to overfitting, it certainly doesn't preclude it.

>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

It is comforting that someone agrees, to this day this is considered a gray-area (or not even that) by students, graduates and those that look up to the University of Buenos Aires institution.

well not if its an open textbook exam

in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc

this is one such case

--

the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute

I think you can do whatever you want with your train set, what would be concerning is training with the test set.

I don't think overfitting is separate, the consequence of putting the benchmarks in your training set is not that you "cheat" by breaking some moral code, it's that it breaks the purpose of the benchmark and trains your model to be good at that benchmark only, instead of being generally useful.

Reply on news.ycombinator.com