Because they are energy dense and not particularly satiating. A slice of bread can be 75-100 calories, a bagel 200-250 calories, a bowl of pasta 400 calories. I like to translate my calories to miles. A mile walked or run burns ~ 100 calories. So that bagel is 2 miles, that bowl of pasta 4 miles.
I'd rather eat fruits and vegetables, yogurt, cottage cheese, eggs, fish, etc.
To be clear, I don't avoid grains entirely. I'll have a slice of toast with 3 eggs at breakfast, or put homemade granola in my yogurt, etc. I eat rice with sushi or chicken tikka. Challah is one of my great weaknesses. It's just that grains are so readily available and easy to eat too much of, so I limit them when I can.