There's a prompt after you hit buy where it requires you to either double-tap the side button (on iOS) or TouchID (on MacOS and some iOS devices). On that prompt, it shows all the transaction information including shipping info, the card Apple Pay will run the transaction on, and the contact information for the transaction.
You must have been speeding through and missed it.
I'm supposed to enter the address on my watch?
You didn't pay attention.
I have shipped to other addresses than my main address on file with Apple.