They will always try to come up with something.
The example provided was a poor one. The comment from LLM was solid. Why would you comment out a step in the pipeline instead of just deleting it? I would comment the same in a PR.
For structured outputs, making fields optional isn't usually enough. Providing an additional field for it to dump some output, along with a description for how/when it should be used, covers several issues around this problem.
I'm not claiming this would solve the specific issues discussed in the post. Just a potentially helpful tip for others out there.
Query -> Person Lookup -> Result-> Structured Output `{ firstName: "", lastName: "" }`
When result doesn't have relevant information, structured output will basically always output a name, whether it found the correct person or not, because it wants to output something, even if the fields are optional. With this example, prompting can help turn the names into "Unknown", but the prompt usually ends up being excessive and/or time consuming to get correct and fix edge cases for. Weaker models might struggle more on details or relevance with this prompt-only approach.
`{ found: boolean, missingFields: [], missingReason: "", firstName, lastName }`
Including one or more of these additional text output properties has an almost magical affect sometimes, reducing the required prompting and hallucinated/incorrect outputs.
It takes less effort to re-enable if it's just commented out and its more visible that there is something funky going on that someone should fix.
But yeah, even if it's temporary, it really should have the rationale for commenting it out added... It takes like 5s and provides important context for reviewers and people looking through the file history in the future.