The physics of interacting with your previous self are undefined with many plausible solutions.
It is entirely plausible that the universe limits your interactions with your past causal self to a small range, allows arbitrary overlapping world lines, blocks all causal violations entirely, or has disastrous consequences for time loops.
My personal bet is that the universe has no reason to care that you've interacted with your past self given a consistent proper time worldline. This however removes conservation of energy and momentum in the universe, which General Relativity also does not preserve.
Of course this does allow for misuse of causal loops to generate arbitrarily large/infinite amounts of energy without any bounds on observability.