The issue is that there is only a limited set of things to do while waiting for someone. I can't get down to work to solve any problem of the day because I may be interrupted at any moment by the person arriving. Then I have to sacrifice my time (and perhaps my other plans) after the meeting to get my work finished, perhaps at a time when I figured the meeting would already be over and my remaining work already done.

I really think it's very disrespectful.