If the NVIDIA DGX A100 won't restore your faith in cool computers then nothing will. https://www.youtube.com/watch?v=TJcKYUTaBtg
These things were the very definition of walled garden.
According to https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo... GPT-3 was run on V100s, which are the previous generation server GPUs ("Volta" vs "Ampere").
"GPT-3 uses half-precision floating-point variables at 16 bits per parameter... At around $130,000, DGX-1 is short on VRAM (8×16 GB), but has all the other components for a solid performance on GPT-3." (DGX-1 is the previous chassis NVIDIA offered for hosting V100s.)
So I tentatively think the answer is yes, a single DGX-A100 probably can run GPT-3 comfortably. You'll still want a cluster of them to train it, though, because they note training takes about 355 V100-years.
In principle you can of course run any model on any hardware that has enough RAM. Whether the inference performance is acceptable depends your particular application.
I'd argue that for most non-interactive use cases, inference speed doesn't really matter and the cost benefit from running on CPUs vs GPUs might be worth it.