I mean, sure, it’s easy to say interesting things if you’re not being held to any standard of rigour.
Ditto for AlphaGo, AlphaZero, and most other AI systems.
If one wants to argue that we shouldn't be doing AI science this way, then that's fine. It's just a different conversation.
GPT-2 (fully weighted) was arguably reproduced as well, eventually 'forcing' OpenAI to release the model they had held back [0].
0: https://www.reddit.com/r/MachineLearning/comments/by4n8p/p_s...
I admit that it's possible the entire world is currently "doing AI wrong," though. T5 and other new projects have been better with respect to replication. But many datasets are still locked behind paywalls, or only accessible if you have an .edu email address.
Personally, I struggle to see what new thing we learned from GPT-2. Did we learn something about the physical world? About how human minds work? About how language works? It's a language model after all. All we learned is that throwing a large dataset to a hard problem can produce results that are difficult to evaluate.
"Science" means "knowledge". If we haven't learned anything new from GPT-2 then it hasn't contributed to science. It's impressive, like a jetliner is impressive, or an aircraft carrier is impressive, but it's not increasing our body of knowledge about the world and ourselves.