I've turned off memory features in everything because of this kind of problem. It makes models dumb as hell. And, it makes them more effective sycophancy machines, which are probably unhealthy to interact with.
I doubt the OP is actually the Gemini system prompt, but I'm sure it does try to keep personal data from screwing up results, but it's just not possible given the state of the technology. Everything you cram into the limited context probably either helps or hurts the results and if it's unrelated to the specific problem, it hurts.
When the model tries to satisfy everything it remembers about me, it comes up with conflicting details and desires. My personal projects don't look anything like my work projects. My little games don't have the same requirements as my security sensitive software for robots in hospitals. The fact that I asked how a hospitality business operates doesn't mean the tax question I asked a week later is about a hospitality business.
The models just can't make sense of all that data yet, and even if they've been instructed to consider that maybe some details aren't important, it still impacts the attention math.