In this sense, a calorie is not a calorie - but to be precise it's more that different sources of calories have different side-effects on the body when converted INTO energy/calorie.
Now to grains (specifically wheat-based products): gluten. The single reason why people in my athletic sphere (CrossFit) avoid wheat products as a source of calories when training is because of gluten (and sometimes yeast - as it does stimulate candida growth).
Gluten can cause all sorts of problems that aren't related to the calorie intake - diseases, obesity, inflammation, etc...
(I won't cite anything here for that because the information is widely available and in many books - although there are still "studies" that would say this information is bunk)
I consume around 4-5k calories per-day as part of my training. As long as you use it, the calories aren't bad for you but the source of the calories must be clean.