You're accusing him of having a problem with it, which his comment does not imply.