Calories do matter, but it is not nearly as simple as you're trying to make it out to be.

Genes play a huge role. For most people, playing the game of watching what you eat and counting calories makes sense, but for others, it's not so clear cut. For example, my dad (a molecular biologist) is studying the effects that certain genes have on the weight of mice. The details escape me (and aren't relevant here), but the takeaway is that your genetic makeup will predispose you to weighing a certain amount, and some mice/people will be fat even if they eat very well, and others will be skinny even if they eat very poorly.

Hence, for some people (but not everyone), managing their weight requires more than just counting calories.