The basic reason is that your body can increase or decreased its energy expenditure drastically in response to increased or decrease consumption of calories.
I'm reading The Obesity Code by Dr. Jason Fung right now and he presents a very strong argument that the cause of obesity is insulin resistance.
In the book he cites both overfeeding studies where subjects were fed very large numbers of calories but did not gain weight, as well underfeeding studies where subjects ate only 500 calories a day but didn't lose weight.
After reading this I'm pretty damn convinced that the diet problems facing us are: highly refined carbohydrates, sugar and sugar substitutes, and alcohol (I haven't come across this in his work, but if you look at the stuff by Dr. Lustig you'll see sugar and alcohol are incredibly similar metabolically)
Think about this: for most of human history we did not count calories and yet we were not obese.
It's the highly processed western diet of breads, cakes, cereals, soda, etc that are the culprit.