I feel like you don't know that you're talking about. That's not meant to be an insult but to highlight the fact that "feeling" doesn't matter. Because my feelings about your statement has no bearing on the accuracy or content of your statement.
"I feel like" in this context is a synonym for "it is my belief that," so your response is a non sequitur based on a misunderstanding of a colloquialism.