This article mixes both “should research be open” and “is this work cheaply reproduced / accessible”:
For smaller, open models:
> The average person could not recreate models of this size from scratch, but the models can run on a single machine with a single GPU.
but about GPT-3:
> GPT-3 represents a new circumstance. For the first time, a model is so big it cannot be easily moved to another cloud and certainly does not run on a single computer with a single or small number of GPUs. Instead OpenAI is providing an API so that the model can be run on their cloud.
While I’d quibble with “for the first time” (it’s easy to generate mega models! Plenty of distributed mesh tensorflow stuff does that, etc.), I don’t think this is any different than large physics simulations.
Is it “wrong” to have some groups push the boundary of what’s possible with supercomputers? I certainly don’t think so. If anything, it shows what’s possible and others can do the valuable work of “miniaturization”. In this specific area, PRADO is a good example relative to BERT. For my historic area of ray tracing research, we did lots of things on an SGI Origin that let us “jump ahead a few years” versus what we could have done on any basic workstation.
You could argue that it’s not academically interesting (“you just ran this really big because you had the budget / hardware, whatever”) and reject the paper. [Edit: I consider this kind of work interesting from a systems perspective, not “ML”, but it’s still interesting!] I don’t think it makes sense to suggest that we should hold back progress based on NSF grant funding or least common denominator computing resources. How would you decide what’s acceptable? Is a single A100 affordable? Only a T4? Only a laptop?
tl;dr: it’s fine to argue about openness and democratization being hollow marketing words. I’m not sure I would conflate openness with “everyone, everywhere should be able to run any scientific work without expense”.