After looking more into it, Claude DOES give the correct answer, just not in the format that it's asked, it always adds more info at the end, even when asked to just give the answer...
It was just an example though, in real-world scenarios, sometimes I have to tell the AI to respond in a specific strict format, which is not JSON (e.g. asking it to end with "Good bye!"). Claude is the one who is the worst at following those type of instructions, and because of this it fails to return to correct answer in the correct format, even though the answer itself is good.
personally it's a blurry line. most times i'm interacting with an agent where outputting to a file makes sense but it makes it less reliable when treating the model call as a deterministic function call.