Technically yes, a “transaction” is on-network - but one can trade wallets (eg: trading paper keys or “Physical bitcoins”). The definition of the word “transaction” is the important bit here.
Edit: Additionally, as someone else pointed out, you could have kept your own copy of the key and could empty the wallet before me.