Shots fired?
It would be interesting to see how far "clean data" can go on the scaling laws.
P.S. A fairly basic website otherwise, but it unfortunately seems to be hacking scroll for no good reason.
For example, the Apache 2.0 license requires in just 4.c:
You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works;
I find it morally unethical that they (likely) just ingest IP of all open source repo's without asking, but also importantly without any attribution.
Let me also note that I'm not against LLM's in general. But I do think training on open source must be opt-in, and I look forward to a world with actually ethical, and traceable (i.e. on what they were trained on, like a bill of materials (BOM)), models.
> without distillation from third-party models
sounds like zero unless they are lying.
Though this is largely impossible these days, unless they pre-trained on pre-AI era data.
Adding "unless" to a statement makes it vacuous if the latter clause is weaker than the first clause. I find it hard to believe that a company willing to violate licenses would have scruples about lying about it.
I think that's the point. "How do I say they're lying without outright saying they're lying?"
It's a common rhetorical trick.
Also, “Microsoft is lying” is not a logically stronger statement, because they might be lying about something other than whether they distilled or trained on AI output.