I think the biggest issue is that high (saturated) fat is often combined with high carb (refined sugars, starchy foods etc).
My diet consists of high fat / low carb foods and it simply works (for me...). I've lost weight (without trying) and my energy levels are much more predictable and stable.
I never watch how much I eat because my body will naturally stop be from overeating.