This is called turn detection, and there are some great tools coming out to solve this recently. (One user mentioned Livekit’s turn detection model). I think in a years time we will see dramatic improvement.