Those are not small delays. The speed of light is slightly faster, so having someone at the front indicating time keeps everything together.
The pointing and gesticulating aren't limited to prepared rehearsal notes. Rehearsal notes certainly happen, but apart from keeping time, the conductor's job is to define the mood and emotional valence, moment to moment. This may be somewhat improvised, within limits.
Orchestras typically play somewhat behind the indicated beat because it allows for more expression.
There's also a lot of business-related admin - dealing with absences, holding auditions for new members, contributing to program scheduling, reading and possibly writing feedback notes to/from various interested parties, and so on.
Every once in a while a smaller orchestra will try to manage without a conductor. While the results are usually workable - professional musicians are very good at their jobs - they're not great.