Isn’t GPT4 a MOE of approximately 16 111B parameter models? Could you not build a MOE on par with GPT4 by building a cluster of models in the range of 16 or so and do a little MOE ?
MOE models are trained all at once, they're not simply ensembling already trained models. Also data quality and quantity matter considerably, and how OpenAI gets their data is not public
You'd have to get a separate license from TII to host the models for commercial use.
I’m far enough in my career and life that LLM for commercial purposes seems very limiting. I can also see a non profit LLM service, sort of like “open ai,” but actually open, and hosted for non commercial purposes. Sort of like a Wikipedia model GPT4 scale AI.