Not all your data, just the data exchanged with that person.
But how could you force this? They could always copy it or whatever. Yes, I know, defaults.
But this would then need to be somehow enforced on the other side, right? Like preventing copying, or screenshotting, etc. And in that case, see the HN thread the other day about applications messing with these functions and how people want their devices to be theirs and not controlled by some third party.
You gave the answer yourself. If the person you are writing with has E2EE enabled for message backups then just exclude that chat from backups on your end or enable E2EE for that chat. That means if you take the restore route without having access to the keys that conversation will not be restored.
You could even let people overwrite that setting. Preventing screenshots or copying has nothing to with this. Apple is claiming that message are end to end encrypted when in reality their defaults are set so that they are not. That is a lie and al they would need to do to change that is to change those defaults.