Comment by nsagent | Hacker News Reader

Comment by nsagent | Hacker News Reader

Parent Full thread

I'm unwilling to speculate whether or not OpenAI is breaking their agreements (I honestly have no clue), but as an NLP researcher I'm certain they could launder data by having an LLM rewrite it and subsequently train on the rewritten data.

Papers like "Curated Synthetic Data Doesn't Have to Collapse" [1] and "How to Synthesize Text Data without Model Collapse?" [2] demonstrate it's possible to do this.

Since OpenAI's Privacy Policy [3] explicitly allows for the use of deidentified data, it's possible they consider rewrites (maybe paired with a model used to identify explicit PII) to be deidentified. Whether OpenAI's legal team thinks rewriting in this way technically means they aren't training on your data isn't something I'm able to comment on.

Here's the relevant Privacy Policy statement:

  We also aggregate or de-identify Personal Data so that it no longer identifies you and use this information for the purposes described above, such as to analyze the way our Services are being used, to improve and add features to them, and to conduct research. We will maintain and use de-identified information in de-identified form and not attempt to reidentify the information, unless required by law.

Please note all the hedging words I used (maybe, possibly, etc). I honestly have no clue if they are doing this. I'm merely elaborating on a possible loophole like you asked.

[1]: https://arxiv.org/abs/2605.07724

[2]: https://arxiv.org/abs/2412.14689

[3]: https://openai.com/policies/privacy-policy/

It's exactly what you've said. I'm speaking about Anthropic though, I have no idea whether OpenAI does it.

> Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs. Anthropic disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs. Anthropic may not train models on Customer Content from Services. “Inputs” means submissions to the Services by Customer or its Users and “Outputs” means responses generated by the Services to Inputs (Inputs and Outputs together are “Customer Content”).

This is the only commercial ToS clause about how they handle your data for subscription users. They only promise not to train the model on your exact input and exact output. There's nothing about not washing your data - the "clean-room" approach, which is obviously easily automatable by a company that specializes in automation. That is not training a model on your data, it is using a model to create derivatives of your data, then training it on "their" derivatives.

People really needs to apply pressure and start demanding answers from these companies regarding this - because it is a huge problem. Historically, the amount of labor required to do something like this would make it entirely unfeasible, so this is all new territory. The existing laws and the requirements around clarity surrounding these conditions do not reflect the technology progress.

That's in fact also very technically feasible. They're already running small classifier models on both your inputs and your outputs. These models are said to be in place for safety, but they could answer 2 questions: (1) is this a safe query, and (2) is this query about X, Y, or Z topics (which happen to be high alpha / very interesting to Anthropic).

If #2 returns a yes, the input/output are temporarily stored (say, for 30 days) while a background process cleans up PII and generalizes the prompt to be "clean", then the clean prompt that is not yours anymore is stored aside and used for training, product development, etc.

Just to be clear, you’re speculating, right? They could also not be doing this.

Speculation is a spectrum, and there's a probability attached. Consider these companies make it a rule to shit on IP as a concept.

If anyone, anywhere, would be doing this, it would be Anthropic and OpenAI. That doesn't mean they are, but that, to me, certainly pushes it past simple speculation.

I totally get the speculative aspect. However, the entire premise of them making money off software engineering is predicated on a truthfulness of whether they use that data or not. Otherwise the Chinese models (which make no promises and do log) become more viable…

I don't necessarily think it is, because of regulatory capture. We're seeing Anthropic and OpenAI try to squeeze the chinese in the courtroom more and more, and I think it's to your point: they're all doing the same things, so now they need to employ extra-business tactics to get their marketshare. I think they're hoping that Chinese models become unviable because they're literally illegal, which I don't think is very far off from happening soon. Which is bad all around, but everyone is playing dirty right now.

Just to be clear, this is not speculation, it is fact.

It's absolutely speculation (unless you have specific inside information) to claim that they are doing this. It's a fact that they could be doing this.

Edit: From your other comments, you seem to be heavily insinuating that you do in fact have inside information. If you do have such information, it's not clear to me what consequences you think you are avoiding by adding this flimsy veneer of supposed deniability.

Yeah, I guess that's possible. Anthropic has published some information about systems they use to monitor usage patterns, which indicates they have some related infrastructure set up.

https://arxiv.org/abs/2412.13678

Side question to you, considering your occupation:

Could you please list a set of core papers (or other resources) that give a beginner an overview or even understand of the fundamental concepts and techniques with LLMs?

Thank you!

I'm sorry, but I'm not sure what works best for a beginner. I started my PhD when the original Transformers paper [1] had just been released. I had no background in NLP whatsoever and used the original paper to write a Transformer and the full training pipeline from scratch during the first couple months of my PhD without referencing any existing code (only reading the paper and it's references).

So I'd say, if you're motivated you could do the same. That said, I've always been a self-starter and I started my PhD after working for a decade. I'm sure there are other resources out there, but I'm not equipped to say what's best for a beginner (I found the original paper to be excellent, but most everyone during my PhD, including my advisor, found it to be inscrutable; I think it's written more like an engineering focused paper, which might be why researchers found it difficult to grok, but with my previous industry experience it seemed quite clear).

[1]: https://arxiv.org/abs/1706.03762

Reply on news.ycombinator.com