In my experience, not so much? You'll need to worry if you're working on tasks that touch the same corner of the codebase, but larger teams tend to correlate with larger codebases, meaning that's not necessarily more likely.

That sort of XP style development also tends to come hand-in-hand with near-constant pair programming. So even if the conceptual limit is "5 sets of committers", that's still double the actual team size. And in practice, the cross-pollination that comes with swapping pairs every day or every few days means there's less of a cognitive load to needing to be aware of what's going on, since chances are either you or your pair already has context.