Sorry, but saying "it's stupid to do X" really is basically the same as saying... "people who do X are stupid". Or at least "they are, in that context, being stupid".

Either way, you said what you said -- and presumably you meant what you said -- so there's no point in trying to wiggle away from it.