I don't think that's true at all, and it makes the rest of your argument a lot weaker.
What he has said is "if someone goes to the trouble of doing the math, at least take a look at it!" because there are a bunch of people who haven't bothered to refute the math in the slightest, but who are demanding retraction nonetheless. I don't think that position is entirely unreasonable, either.