Oh - I 100% could not care less regarding the morality/legality/whatever... Everyone trains on everything.

I'm just wondering if the smaller labs see the same velocity of advances without SOTA models to generate Terabytes of training data?

Hard agree.

The question is - if the SOTA model disappear - do these follow-on models have the ability to improve themselves without distillation?

> The frontier models stole the whole internet

What does that even mean?