Metabolism is controlled in a large way by hormones. For example, extremely undernourished people will slow their metabolism way down in order to preserve what fat reserves are left. A diet that significantly increases blood sugar will, in many people, cause a high insulin response and cause fat to be stored and only glucose to be burned.
Essentially you're looking at this from the wrong way. For the body to grow in any manner, hormones are required (c.f. growth hormones during puberty). And for the body to have grown, excess calories must have been consumed. What forces people to eat so many calories that they grow in size is triggered by their hormones generally telling them that they are starving, even if they are not. See Metabolic Syndrome, Diabetes, Insulin Resistance, etc.
So yes, calories in/calories out, but you're not even considering _why_ those calories were consumed in the first place.
Disclaimer: I've been eating Low-Carb/High-Fat diet that brought my steadily worsening weight issue to a much more healthy level (91kg to 75kg) through control of my blood sugar and appetite.