That's not really the case. For many people, if they know that too much sugar is bad for them, they think of sugar as in white sugar they add to the coffee or in terms of candybars, when instead they should think about carbohydrates as a whole. Replacing soft drinks with orange/apple juice many people would consider a healthy choice, but in the end you're still trading sugary water for sugary water (unlike eating whole fruit which has fiber and other elements to balance out the fructose). Then you get into potatoes, pasta, rice and bread which don't help when trying to lose or maintain weight.
Ask around amongst your friends and family. Tell them that you're trying a low fat diet. They will congratulate you on your life choices. Tell them you're trying a low carb diet. They will tell you you're killing yourself and you should stop because body needs energy to survive. Most "diet" products replace fats with carbs, which may even be a net negative in some cases when trying to diet.