Architecture as in you may have multiple GPT instances being prompted to look at a problem from different angles: "Analyse the problem as a pessimist", "As an optimist", "as a mathematician", "an engineer", "a philosopher", etc.
Caching, as in determining what you can store from these outputs, e.g. "give a numeric 1-10 score of what you think of this product as a LGBT-friendly conservative from the Midwest", etc.
Sure, but just as a thought experiment imagine you have a really bad disease. I give you two options: 100 high-school students can diagnose and prescribe a treatment or you can choose 1 professional with 10 years of experience in related diseases.
You might initially prefer the professional with 10 years of experience. Would your opinion change if I told you that I selected the high-schoolers to be diverse so that one is pessimist, one is an optimist, one got good grades in engineering, one loves philosophy?
Of course, my intuition might be wrong. For example, perhaps it legitimately would be better to have 100 high-school students where one is a high-school level ability with ear-nose-throat, one is a high-school level ability in oncology, one is a high-school level ability in cardiology, etc. Except some control AI would have to synthesize their answers into a coherent response ... and that controller would be high school level.
I'm not really sold either way if I am honest. I don't think we have the answers to these questions. It just shows that my intuition about intelligence is open to challenges.
Meanwhile, the "100 high school students" on a mobile phone are the only medical consultation someone in Sub-Saharan Africa or Southern Asia is going to have access to at all.
"Why Not Both" would surely apply here. Maybe we can't give everyone access to a kind, caring, patient human doctor; but we sure can make coming in second place a lot less painful.
[0]https://www.who.int/news/item/13-12-2017-world-bank-and-who-...
What I mean to say is, if I am considering building a product based on LLMs then I may have to make a basic decision: can I use multiple cheap LLMs in a multi-agent setup or must I use a single expensive powerful LLM. Right now I don't have any intuition on what kinds of problems are solved most efficiently by either approach. Just looking at a problem description I can't intuit which approach is appropriate.
If you try to start with the multiple cheap LLMs approach, it is almost certainly going to be more difficult to get the prompting right, and if you don't already know what you are trying to accomplish is feasible, you're adding a lot of work that might add up to nothing (even if it would actually work if you used the more powerful model).