Get the dataset minus the illegal data back. It's essential to maintain parity with closed source efforts.
I'd bet that OpenAI also has this in their dataset too. And nobody will ever even know. Open source is the only AI that can be knowingly scrubbed and cleaned.