How are you measuring “rightness”? Does it have to exactly match the predefined JSON output? Or is there some qualitative “sameness” check going on?

If you’re not already, you could probably give GPT-4 the correct JSON and the predicted JSON and ask it whether the answer is qualitatively correct and assign a letter grade much like a teacher would grade something qualitative like an essay.

The “parse the graph” one seems almost correct. It’s labeled as “Failing” which is correct if failing means “not perfect” but I’d probably give it a B- rather than an F.