Reducing Doom Loops with Final Token Preference Optimization

It would be great if this could be combined with quantization-aware finetuning. In my experience, Qwen3.6-27B has much fewer repetitions at Q6 quantization level as compared to e.g. Q4, but that leaves little space for context on my 24GB RTX 3090.

Somewhere around the initial ChatGPT release, I was experimenting with having the model draw various types of ASCII art. I believe the first time I ran into a doom loop might have been when I asked it to draw a giraffe; it started with a head, then a line of | | for the neck, and then it just repeated that, ad infinitum. I am not really sure what the cause was, but clearly it wanted to make a long-ish neck, and ran out of a way to determine how many lines it had already produced, so just kept going.

> On an early checkpoint of LFM2.5-2.6B, 10.2% of completions on hard math and coding prompts produced repetitive loops. After Antidoom training, that rate fell to 1.4%, with eval scores improving across the board as a direct result of reduced looping.

That's a 9% reduction, and it doesn't really discuss applicability over longer contexts, different language/language familes/programming, or context/rope length aspects.

Several families - esp Llama and gemma - seem to be easily doom-inclined during expansions (I realize this is a pathological misuse of the tooling, but it's what people do): prompt: "add comments to this file", +1000t file read, [+3k reasoning], +800t edits, +1100t file re-read promoted by tool, prompt:"pep8 format it", [+2kt reasoning], +700t edits, +950t file re-read.

The unmodified/repeating sequences in those file reads, the differences between them, is a honeypot for attention.

I'd non-scientifically concluded for myself that this was tied to an anti-pattern for rope. In the first portion of the context inference needs attention to fall in certain places to produce the correct suggestions for the first round of edits, and somehow largely refocus attention to the subsequent version of the file in the context to correctly apply the subsequent edits, not getting distracted by the tantalizing pattern of repeated sequences and disrupted sequences

Using Gemma4 e2b I can repro this in LM-Studio, and fix it by hand-editing the context so that there's only the latest version of the file being worked on up at the start of the context, or replacing the original reads with something like:

" [tool name='read-file' path=...']... elided, see file.v1.py if you genuinely need to revisit the original state...[/tool] "

The handful of times I've tried this over the last 3 years it hasn't cared to discover file.v1.py didnt exist, limited samples = limited confidence.

I'm curious if this approach can be generalized beyond "doom loops".

For instance, another way of thinking about a "doom loop" is wasted tokens, which happens all the time with larger models that are inefficient at test time. Can "bad-ish" tokens be identified and penalized?

Maybe this is already SOTA but would love to learn more!

this is pretty cool. i think part of the root cause is current rlhf post training design around confidence and optics rather than cooperative transparent honesty. though its kinda an expensive hypothesis to dig into as a private individual

Interesting approach. From the practitioner side, the doom loop pattern I hit most with coding agents is context pollution: after two or three failed fix attempts, the failed approaches stay in context and the model keeps circling them. Clearing the session and rewriting the prompt with the exact symptom almost always beats a fourth correction. Curious whether penalizing at the token preference level actually reduces that, or whether it mostly shifts where the loop shows up, since the root cause often seems to be the conversation history rather than the model's token-level tendencies.