This is one of the many benefits to working for a high performing manager. They will have more clout to argue on your behalf for raises and promotions. Not to mention it's easier to grow around higher caliber people.
Personally, what I look for the most is honesty. Even if the product/company is crap, if the people working on it are honest about their current state and where they want to be in the future, I find that much more interesting than a well-known established company whose manager is just OK.
pick the team to join solely based on the manager
My experience too. Of course there are other factors, but this is (arguably) the most important factor of all; the rest are just constraints, and this is like the objective function.