Comment by zby | Hacker News Reader

Comment by zby | Hacker News Reader

These sampling based techniques is a rare occasion where experimenting with consumer hardware can let you improve on SOTA models. I don't think it will last - the end game surely will be a trainable sampler. But for now - enjoy tinkering: https://github.com/codelion/optillm implements a few of these techniques

optillm authors suggest that the additional computations in Entropics don’t bring any better results in comparison with the simple CoT decoding (but I am not sure if they also check efficiency):https://x.com/asankhaya/status/1846736390152949966

It looks to me that many problems with LLMs come from something like semantic leaking, or distraction by irrelevant information (like in the GSM Symbolic paper) - maybe there is some space for improving attention too.

I wrote a couple of blog posts on these subjects: https://zzbbyy.substack.com/p/semantic-leakage-quick-notes, https://zzbbyy.substack.com/p/llms-and-reasoning, https://zzbbyy.substack.com/p/o1-inference-time-turing-machi...

The problem that I see with all these different sampling techniques is the way people usually judge them. There are people who claim they work better, but no rigorous benchmarks to prove it. Lots of "it writes better" or "the prose is fresh", but that is one argument where I think LeCun is 100% right - you can't judge a generalist model by "it works on poetry" or "prose", because that's the definition of bias, and you're shooting yourself in the foot with personal anecdotes.

I'd like to see this applied to coding or math. See the samplers work better in say olympiad math problems, with thorough benchmarks before and after.

The min_p paper and many other papers are doing exactly that.

Is this [1] the paper you're referring to?

Unless I'm reading Table2 (page7 - pdf version) wrong, on math, min_p is shown to score worse than top_p.

For temp 0.7 it scores 1 point lower than top_p. And from temps 1.0 and up, while scoring higher than top_p for the same temp, it scores way lower (6points and up) than top_p at 0.7. So overall, if you want accurate answers (and for math you kinda do), min_p is worse overall? Unless I miss-understand something.

I agree with the authors that if you want a tradeoff between accuracy and diversity, min_p might help, but if you're looking for precise answers, the results will be slightly worse. It's a tradeoff, but as I said above, people often fail to mention it as such, and instead proclaim it to be "better" across the board.

[1] - https://arxiv.org/pdf/2407.01082

If the objective is to make a better poet or a better story book writer than this flawed metric is the only form of measure.

It’s the same measure we judge human writers on so it’s not necessarily the worst.

Semantic leakage could be just weakness of the model, and related to claims that they don't _really_ reason. Maybe more training could help.

Or maybe it's a more fundamental weakness of the attention mechanism? (There are alternatives to that now.)

This is incredible! I haven't seen that repo yet, thank you for pointing it out, and the writing

Reply on news.ycombinator.com