Your situation sounds quite like any other big corporate. Lots of people not doing anything or working very little and manages that can be really picky and hard to get along. I am not sure what I would do, getting into that team seams like a way to go, but working with that manager sounds very difficult. Maybe give it a go for few months to see if you get along with him? If not, start looking for a new job, while you are still at your present one maybe.
It really depends on your character, I might not give the best advice cos I am tired of working for ineffective managers.