Hacker News Reader

There's no difference in the output distribution between always doing constrained generation and only doing it on the failures though. What's the advantage?

View on HN

There's no advantage wrt output quality, but it can be more economical in some high-error regimes, with less LLM calls used in resampling (max 2 for most errors).

My point is that if you're capable of doing constrained generation and want to try once and the constrain on failure, since that has the same output distribution as doing constrained generation in the first place, you'd be better off just doing constrained generation always (max of 1 LLM call for the class of errors fixed by this).

There's only a different distribution with 2+ initial attempts before falling back to constrained, at least if I haven't screwed up any math.

Reply on news.ycombinator.com