A really bad colleague can make a job miserable. I slightly bad manager can also make it suck.
For the 2 jobs I had before going into consulting (working full time for consulting companies) where I was a strategic hire, one of my requirements were that I reported to the director/CTO and not the line level managers. Someone who could actually sign off on a decision.
In consulting it doesn’t matter. The client drives the decisions.