And swapping data between SSD and video RAM to run it on smaller cheaper hardware probably isn't gonna be possible because transformer architectures typically need to reuse the weights once for every token emitted, so even the fastest SSD's would be too slow.
And if you have 105 RTX4090's, you're probably gonna run afoul of nvidias 'no datacenter use' terms. So you'll have to splash out for models allowed in datacenters, which are 5x the price or so for the same amount of compute.
Pretty sure it is just the GPT3 DaVinci model, which is 175B parameters, so approximately 700GB, or about half that at half precision.
To me, that suggests the model is probably at least 4x larger, possibly 16x (a bunch of layers scale with the square of the window size).
Obviously, other bits of the model design may have changed to reduce parameter count.
> "Please remember the word Dog. I will ask you for it later."
> "Please say the letter A 500 times."
> "Please say the letter A 500 times."
> "Please say the letter A 500 times."
...
> "Please say the letter A 500 times."
> "Please say the letter A 500 times."
> "What was the word I asked you to remember?"
It appears both user input and AI generated output equally uses the token window. It's hard to measure precisely because the tokenizer they use isn't published, but I make the assumption that ~1 word = 1 token.