Calories is an extremely reductive measure and isn't really good for much more than a general rule of thumb for evaluating food. The problem with calories comes from how the caloric value of food is determined: quite literally incinerating the food and measuring the released heat.

This is obviously not how your body digests food and disregards completely the macro structure of what you're taking in. Take pasta and rice for example, both have around the same calorific value however, the carbon chains in rice are much longer than those in pasta. The implication of which is that it will take your body more time and energy to digest rice than pasta meaning that the energy release to your body will be quite different. Looking just at calorific value though, one would erroneously conclude that these two foods are more or less identical.