I don't think in your case it would be a lie to say "It's not a good fit." Design is a pretty subjective area so it's likely the case someone else will find his work to be higher quality than you do. Maybe he's asking you for critiques because he can tell you aren't digging what he's producing, or maybe the vibe between you is no good and you're perceiving that as mediocre work. In any event, best to get it over with; you'll both be better off without one another it seems.