Dart had quite a long history before Flutter even came into the picture. Flutter is the answer to the question of why is Dart somewhat relevant today, but not to the question of why Dart didn't take off in general during its first 5 or so years, despite the fanfare of coming from Google.
Microsoft, Apple and Mozzilla did't felt like integrating DartVM on their browsers, that is why.