First, note that the calorie estimates on food packaging can be off, sometimes by as much as 25%.[1] Restaurant foods have an even wider discrepancy. [2] And as you point out, different people will digest them and absorb different nutrients from the same food making comparisons difficult. For example, I absorb a small fraction of the calories from dairy products because they unfortunately pass through my digestive system quickly and remain largely undigested.
And as others have noted, accurately estimating calorie expenditure can also be difficult without professional equipment, so in practice measurements of calories expenditure can be off by a significant margin as well.
Combining all of these sources of error can result in excessively wide error bars. Someone who relies on calorie in/out estimates to dictate their consumption may end up eating significantly more or less than is optimal.
Of course, even if counting calories in/out is wildly inaccurate it might still have a positive effect if it causes people to be more aware of the food they eat. Unfortunately I'm not aware of research that teaching people to count calories results in better health outcomes than other diet strategies. (Though it does seem to increase people's odds of developing eating disorders. [3])
It's one of those cases where the theory may be correct in theory but that doesn't make it particularly useful in practice.
[1] https://pubmed.ncbi.nlm.nih.gov/8371446/ [2] https://pubmed.ncbi.nlm.nih.gov/21771989/ [3] https://pubmed.ncbi.nlm.nih.gov/30155857/