Well, some people are trying https://hirokatsukataoka16.github.io/Pretraining-without-Nat...