Interesting experiments. Too bad they're not releasing the JFT pretrained models. I guess the cutoff point of what's too valuable to share has been reached.
JFT is the exception. I find JFT interesting so I pay close attention to anything using it, and as far as I've noticed, no model has ever been released based on JFT, going back to 2015 at least when it was much smaller. It's always either held back or the released model is based on public datasets (eg BigGAN - released G was on ImageNet though the paper notes that the JFT BigGAN completely avoided divergence problems, which is very interesting). I've wondered if legal/copyright issues block any release: there's always someone who tries to argue that a model is a derived work, and nothing in the JFT-300M papers mentions having licenses covering public redistribution.