Agreed. After reading the title, I immediately thought of the Doherty Threshold [1]. It states that if the system can respond to user actions and give the user feedback within 400ms, it increases users’ attention and productivity.
That constant delay, plus the worse ones when the connection loses strength, put a constant toll on "zoom" conversations.
I think that is the main factor but many of the items they brought up in the article also add to it but my guess is if there were no delay or drop in quality, it would be very manageable.