The most basic function of a director in a concert is to dictate the tempo. You can make do without a director in a small group of people by just looking at each other in tricky moments, but when the people involved become dozens or hundreds it is impossible to stay in tempo without a director.

You could take the director out and people would still be able to play, but it would most likely result in a much worse performance.