Were you ever irked by a polite, yet completely groundless compliment, offered out of convention? For example, when someone who is completely unqualified to judge your work tells you "Great job!"
In a situation like that, I often feel two simultaneous reactions, one being "How would you know?" and the other "I appreciate the sentiment." Guess which one I choose to act upon?
Just as "great job" from someone who doesn't know anything about it is not such a big deal because it was simply a nice gesture, so can "too much time on his hands" be (sometimes) considered to be no big deal because there was no specific malice behind it.
In other words, while I agree with the sentiment, people sometimes need to grow a skin bit thicker than they have.