I think Feynman was just jealous :) When you consider that von Neumann was at IAS and he managed a huge amount of collaboration of vast importance I think it had more to do with the individual than the environment.
I do think it is more of an issue with the individual than the nature of IAS as great collaborative work was done by people there.