> What if one model has the exercism solutions in its training data, but the other model doesn't, is the comparison still fair?

I agree that it might not be fair. However, refactoring is a different story imo since as I see it relates to more general capability of the model (description about refactoring tasks are in the link from previous comment).