This is the "oracle attack" that certain parties warned about already years ago. But I'm not sure how relevant the attack is in this context because the data is, in theory, open to begin with. But a good warning about training LLMs with in-house data.