Hacker News Reader

For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code editing benchmarks of GPT-3.5 and GPT-4:

https://aider.chat/docs/benchmarks.html

I’m curious if you have measured whether the “constrained generation” that you’re doing suffers from similar downsides?

View on HN

We’ve seen this too. We run them as two separate stages - “reason”, log the intermediate output, then parse.

100% have observed the same over many tests. No loss in fidelity when responding in spoken language style of formatting but using json is disastrous.

Using OpenAI Function Calls or asking for JSON in the prompt?

I have noticed it in both but have been working with json output before function calling was introduced so I have more evidence on that side. The times I have tried to implement it in a function call I was equally unimpressed with it.

While not ideal, could a workaround be to ask in spoken language first, and then ask to format it in JSON?

That’s what we have been doing. Two passes. The task and then the format.

Reply on news.ycombinator.com