Yes, and LLMs have been shown to store and be able to output their training data, so this is at best very sketchy