To get to this, I asked him to predict the return value of some simple python function (for """def func(a,b): return a*3 + 2*b)""" what's func(2)).
without requiring the explanation, the answers were not only wrong but also changed between attempts for the same input.
Asking the model to provide an explanation, however, resulted in it giving the correct answer each time.
This method still allows for improvement in the quality of answers in more complex problems, although at a bit lower correctness rate.