My mental model of bodyweight is that my body can adapt to use less energy to some degree, probably a few thousand kJ at most. If I exercise or drop my caloric intake and it's still within that range, I won't have to dig into fat reserves much. If I do more than that, I absolutely will.

As you say, doing low to moderate intensity exercise for a long time -- like walking 20km a day for a week, which is double what the Hadza walk -- is enough to cause me to lose weight unless I'm eating more to make up for it.