I can certainly believe that model training cost is exponential as the # of parameters goes exponential. But that is a one time(-ish) cost relative to actually using those models.
Or am I completely off base here?
There’s ongoing research to reduce the computational costs of inference, but to my knowledge they only offer linear improvements (although I wouldn’t bet against more substantial reductions in the near future, particularly as these techniques are compounded).
I hear this a lot, but don't see it very often in practice. GPT and it's ilk are coming up on 3 years old now, and our most novel application thus far is the same textbox + response that Talk to Transformer had. That's sad.
I've heard the sell before ("imagine AI spreadsheets!") but I don't think people are willing to pay for answers that are regularly wrong in the long run. If your bridge only works half the time, people probably won't be inclined to pay your toll anymore.
Until I see something like this rolled out with widespread success, I'm gonna doubt it. The second someone puts an AI agent on their website, it's a race to get the brand to endorse the most abhorrent thing possible. Then what?
Just be clear: the alternative to letting a future GPT-derivative write your novel is to go to the library and pick a book. That works now and has negligible emissions. A future GPT (currently you won't get a consistent long form text) might do the same - optimistically it will do it on only 2-3kWh of energy (which amounts to the cost of creating a physical book). Granted, you only can sustain a human on that for a day, but then the humans don't go away because of AI.
Conclusion to me: the things are fascinating and so are H-bombs...
Question: which kind of time do you expect people to spend with these models? Which human needs will they satisfy? How will their energy consumption then still be negligible?
I bet this has some potential.
That should be order of magnitude less.