The type of food is what makes a difference. Fatty foods are worse than the same caloric-quantity of fibrous foods (i.e., fruits and veggies).
And there's plenty of research to suggest that food in - food burned by activity (inc. exercise) - undigestable food = food stored as fat.