Right, and furthermore Calories In Calories Out is a problematic system anyway.
You stop by a restaurant and order a burger. Maybe the restaurant has calories posted. Those calories are probably based on the "corporate standard" for that item, or might be based on buying samples of the product from several locations and averaging. But your burger won't have exactly that number of calories. Perhaps this person puts more or less of certain toppings on than is average. Perhaps your patty was slightly larger or smaller than average.
All of that means it is hard to know the real calorie count. The same sort of thing will apply but is possibly even worse for food you prepare yourself from scratch. It is not trivial to even calculate the calories in meals you prepare, especially if fresh produce is involved. (For packaged ingredients you can try to go by the info on the nutrition label, but be warned that the rounding rules can make those misleading at times. But for fruits and vegetables, the best you can do is look up the average Caloric density for this variery, and hope that the items you have are not from some local strain with meaningful differences from the normal for that variety). In the end, it is actually easiest to accurately count calories when only eating prepackaged foods, which are generally viewed poorly.
And this is not counting the fact that the systems we use to determine the calories in food are not completely accurate. Using a calorimeter to measure the heat energy output by completely combusting the food is objective, but flawed because the body does not perfectly combust everything we ingest! The Atwater system we currently use can at least in theory account for this, but the numbers used are not especially accurate and have known weaknesses. Plus they depend on accurately determining the number of grams of protein, fat, carbohydrates, non-digestible carbohydrates (like dietary fiber), etc. I suspect that even on extremely tightly controlled packaged food we could be out by over 10%. The limited precision of the Atwater numbers could easily bring the food to a 5% margin of error on its own, before considering accuracy issues with those numbers, or accuracy/precision limitations on determining the protien, fat, carb contents.
And so far I've only tackled the calories in side, and not issues with calories out, like accounting for the basal metabolic rate varying in response to changes to the diet, or that not 100% of all nutritional value will get extracted from the food, and many others. There is a LOT that can be said about that side.
But bigger picture, compliance with any given diet is an issue, and any diet that approach that requires counting calories means more effort must be expended to comply, making it much more likely that the person will either "cheat" on the diet, or just stop even trying after getting worn down enough. That is why even if CICO is thermodynamically sound and sufficent, it is not in and of itself a viable dieting strategy.