Hacker News Reader

In their paper they say "To prevent overfitting, we use Low-Rank Adaption (LoRA) [35] for fine-tuning . . ."

I'm way out of my league here so I have no opinion on whether or not that actually addresses overfitting.

(that quote probably doesn't capture their intention - just a pointer into the paper)

That’s to prevent overfitting on their dataset, it is not to prevent overfitting on the test data, which is likely in their dataset.

You basically cannot beat GPT-4 on broad reasoning tasks, which the tests are designed to cover, without having some of the tests leaking into training dataset. There simply aren’t enough parameters and isn’t enough training to make that possible.

Reply on news.ycombinator.com