I think what the conclusion, though, is that the loss of privacy to language models pales in comparison to the loss of privacy to the rest of the internet.
The language model is just a different way of organizing the information that is already out there. It doesn’t create or expose any new information, and it’s ability to correlate information with other information isn’t superior to the other, existing, ways. If someone is attempting to find out the real person behind your username, a google search still works better than typing your username into the language model.
The real lesson (and one that predates language models like GPT), is that we can no longer rely on the strategy of being ‘lost in the crowd’ to preserve our privacy. When you took all of those actions that connected your username to your real name, you probably thought, “this is just one little obscure thing, no one will ever find it.” Of course, Google finds everything.
This is completely unrelated to things like GPT.