You're completely discounting the effects of fasting on insulin levels. Low levels of insulin are correlated with higher rates of lipid oxidisation.
Not that this effect will outweigh e.g. an additional banana you devour because you're famished after fasting. But it is there. Counting calories is a good help in making sure you don't nullify it.
(https://www.sciencedirect.com/topics/biochemistry-genetics-a...)