This has been my experience working on a distributed team as well. I think my ideal setup involves getting the team together in person once or twice annually for a few days where the focus is brainstorming/collaborating.