LLMs do not have persistent memory. OpenAI does. Persistent memory is nothing magic, it's just LLM context that you, the user, decided to automate its creation to an LLM that can consume 3rd party input. (In fact it's precisely because LLMs are stateless why you could implement persistent memory yourself, completely client side, you don't need any support from OpenAI to do this.)

If you have decided to give a 3rd party control over your LLM context, that's on you. Of course the 3rd party has as much control over the LLM as you do.

It's literally the same thing as running a random .exe from the internet. Of course this can be useful, the .exe could provide a useful function, alternatively it could also steal your data. But you chose to run the .exe. Similarly automating your LLM context generation can be useful, but with exactly the same caveats, whoever influences your LLM context controls the LLM. If you enable persistent memory you give them this control.