You’re right. The consensus on the dangers of sugar is widely accepted in endocrinology and biochemistry. But a lot of nutritionists argue that a calorie is a calorie regardless of the source – they seem to be startlingly/conveniently naive.
Annecdotally I also find that complex carbs like oats may be modestly better for weight than equivalent processed carbs but more importantly they control hunger more. A bit of shopping around will also find specialty pasta that uses a roughly calorie neutral fiber mixed in with flour for pasta that is high in fiber and has half the calories. This too appears to have a significantly different impact on appetite. (Fibre Gourmet is one such brand, texture of cooked pasta is slightly different)
So calories ~ calories for raw body needs of energy, with correlation varying on particular source pairs, but secondary effects are significantly different. Also general human metabolism is still not fully understood, with significant gaps, and anyone claiming to have "the answer" or a perfect diet are at best incorrect.
As a side note, it is difficult to manage hunger at a reduced calorie diet with only calorie-dense food, which makes adherence to a low calorie diet much harder. On average I have found that less calories dense foods tend to be healthier at the same time that I may eat larger volumes of them and feel more satisfied.
Source: Me, a non expert with annecdotal experience and a reasonable amount of non-expert research for balancing my own nutrition & health.