It seems to me that it's not just the extra calories, it's where they come from (sugar and refined flour carbs).
A couple of years ago I stopped eating sugar and bread. I went on a vaguely paleo diet, eating mostly meats, veggies, eggs, some dairy, and nuts, i.e. everything I used to eat minus sugar and bread. I did did not go hard-core paleo by any means. I lost 50 lbs pretty easily though I was not that I could tell eating less or less often. I ate when I was hungry, and stopped when I was full (easier to do, by the way, if you're not eating bread, pasta, or sweet things).
I never counted calories, so maybe I did consume fewer.
Recently I started eating some bread again. I really am a fan of pizza so I thought I would try indulging that every once in a while again. I have the occasional sandwich. Since doing that I gained 10 lbs back. Again I don't feel like I am overall eating more, but in my personal experience those types of carb-heavy foods cause weight gain much more than anything else.