I've worked for brilliant managers in great teams - and left because the business wasn't able to support my growth. Or because the company was ethically dodgy. Or because they just couldn't pay enough.
And, yeah, sometimes I've left because of a bad boss or grim team members - but that's been a rarity.
But, at the end of the day, a job isn't your life. You don't need to find meaning there - you don't even have to like your co-workers - as long as you have a life outside of work.