Flash was a complete security nightmare.
However, as a platform the player was lean and optimized. A lot of people wrote horribly inefficient things with it, which gave it a reputation for being power hungry and slow.
There were additional performance issues on Mac desktops where certain APIs for video playback were blocked.