Rather than the lead programmer, artist, designer, and producer having a good picture of the development process, the project manager was the only one who had a view on where we were in the project. He was not from the games industry which made it even more difficult.
We were only using a subset of agile, but it seemed to break up the team rather than get them working together.
I'm not saying this is the fault of the agile method, or that agile is wrong for games development, but in my experience it was enough never to want to use it again.