You assume incorrectly. I flagged your comment for breaking site guidelines, not downvoted. Perhaps you're unfamiliar with how the site works but it's not possible to downvote a reply to one's own comment. The button isn't there.