I got married a few months back, and decided to eat sugar again from the day before the wedding until the end of the honeymoon. One week in to the honeymoon I'd noticeably gained weight, so cut it out again for the second week ... and ended up only half a kilo heavier at the end.
And then I gained 30 lbs of that back by moving to a new city and letting myself go. Physical activity did not really change.
What goes in has way more to do with your weight than what goes out.
Good books on diet:
In Defense of Food, Michael Pollan
What To Eat, Marion Nestle
Eat, Drink and Be Healthy, Walter Willett
Mindless Eating, Brian Wansink
Why We Get Fat, Gary Taubes
My suspicion: heavy exercise gives you a hunger response in excess of the calories you've burned. That said, I've been told your basal metabolic rate goes up with the more muscle you have.