When a conference call is made up of people all in the same city on decent internet connections, latency is usually not a big issue.
But when a conference call has people from New York, San Francisco, and Japan on it, even if it's only 3 participants, latency can be bad just because of the speed of light, essentially (on top of what is otherwise reasonable hardware/software latency). Latency may be bad even if you're talking with a colleague in the same city, since the audio is "mixed" on the server, and that server might be across the world if a participant from across the world started the meeting. (Counterintuitively, the latency with your local colleague could be twice as bad as with the colleague from across the world.)