You can't know that for sure. I could pay you 2 BTC by giving the private key to a wallet holding 2 BTC. You could even return change in the same way.
The physical Bitcoins (like those by Casascius) work exactly like this: physical coins containing a private key hidden behind a seal.