The Gemini benchmarks appear to be heavily gamed. They seem to be carefully presented and use different prompting techniques to show Gemini beating GPT-4. Since no one can actually use Gemini Ultra to confirm, I think it’s safe to assume this is fluff for shareholders and that once the public gets hands on it will be apparent that GPT-4 is still the better model and better aligned with fewer hallucinations.

There’s very little information on how soon we might see GPT-5. OpenAI seems to have thrown the kitchen sink at GPT-4, so without any new algorithm breakthroughs (which they are rumored to have made with Q*, but that might be just PR) I don’t know that we should expect miracles.

I’m bearish on LLMs in the next 1 year but bullish over the next 5 years.