There is something that I noticed quickly:
The fatigue depend of the quality of the call.
If someone in the call have a lot of background noise, have noisy environement, or a bad microphone, it takes a lot of mental energy to process what they say.