It is not just issue of raw calories but how much body can absorb. Fruitarians for example tend to be corpse-skinny despite fruit being full of sugar, because most of that sugar simply passes through. So effective calories are less than what sugar content would indicate.

But grains and seeds do seem to be quite obesogenic, yes.