It's simpler even: there are conversations that are straight up never happen in writings. So by training only on written text, you will never got those conversations. For example: teacher - student conversation where one side is confused and need to be explained, or many kind of debates and discussions where there is no end conclusion.

Basically, pick a random human from the street and they would be talking and listening a lot more than they would be writing. Those talking and listening is never going to be captured by any digital system.

It might end up that written text has enough similarity with verbal communication that it doesn't matter anyway. But that's hard to guess as a priori that it would be the case.