The point you are missing is that with one diet you would out of control crave far more calories. While on a different diet you would have to force yourself to eat beyond feeling satisfied to reach the same number of calories.
What you eat will of course matter when it comes to nutrition, muscle gain, satiety and many other factors but that in itself does not generally remove that a calorie is a calorie (unless extreme diets).