At least Drupal 8 is much better about this, by default it doesn't load any JS unless there is actually anything on the page that requires it. And scripts are added below the main content and not in head. Caching also got much better in Drupal 8 and is enabled by default, which makes a huge difference in speed.

You always have to be careful with a CMS, it's very easy to load a lot of unnecessary stuff, but you can still build very fast and lean sites with Drupal with only a bit of knowledge.