There's also the question of, if you can distribute a key which is at least the same size as your message over a secure channel - why not just distribute your message over that channel in the first place?
I read some years ago about a non quantum technique to achieve the same based on (I think) noise in a coupled electronic system. I wonder if that has been tested further.
But the question of, "Why not just send the message instead of the pad" is pretty straightforward: when you have the opportunity to safely deliver the pad, you don't know what the message will be. When you do know what the message will be, you don't have the opportunity to safely deliver the pad.
Latency? You can hand deliver a password ahead of time, but not messages.
You still have to reliably move a chunk of out-of-band information in a way such that it gets to (and only gets to) the person you want to have it.
Also, it isn't just a "chunk", for one-time pad it has to be the same length as the messages. Which is fine if just short messages but a lot harder if lots of data.
If can exchange lots of data, better off using them as keys for stream cipher.