> I would feel offended if anyone would feel the need to go easy on me

"go easy on me" means "condescend", so you're essentially doing the same as the people mentioned: feeling offended because they perceive condescension from the speaker.

> Do I look like such a weak person to you that you feel like you have to handle my self esteem with silk gloves?

> Making things up in your head

The phrase he came up with doesn't actually state that he is treating you with silk gloves; that's an implication you're deriving (in your head), just like those people are deriving another implication from the original phrase.