"It's a lemon"–OpenAI's largest AI model ever arrives to mixed reviews
arstechnica.com
arstechnica.com
Chain-of-thought is well understood to be the way to squeeze out performance out of these models. Slow down, go step by step, use more tokens to get a little bit better output. This is so useful, that often in the comparision graphs you see between different models, the best one (the one I want to sell) is using chain of thought while others are not. Not always, but companies have been caught using this technique to hype up their models.
We’ve already seen that CoT (usually!) improves a model’s performance. So does prompting the model with examples of correct question-answer pairs (called in-context examples). But in reports, some models are evaluated with CoT, while others aren’t. The number of in-context examples is often different, and the prompts are almost always different.
https://asteriskmag.com/issues/07/can-you-trust-an-ai-press-...
As soon as OpenAI was desparate for performance upgrade, they implemented this as an actual feature.
The commonly-cited scaling laws[0] predict diminishing returns from scale. Fairly uncontroversial for instance that going from 1 GPU to 2 GPUs gives a larger improvement than going from 101 GPUs to 102 GPUs. Same with, say, computer graphics.
I feel scaling laws are getting conflated with the theory of an exponential "hard takeoff" of self-improvement (which isn't particuarly well-founded in my opinion).
Oh right, it doesn't exist. The only thing that exponentially increases are the costs and no real, sustainable business model is in sight. AI companies have no moat and are ironically consistently threatened to be made redundant.
Businesses keep running around with the solution that is genAI but can barely find any problems to use it on.
We won't be getting what sama promised, that much is clear, I'd say. Thankfully.
I do think it's time to prepare for the post-AI-bubble age. Big changes are coming, after all, hundreds of billions have been wasted for what's essentially a toy.
OpenAI and every other peddler keeps promising upgrades in the magnitude of 100-200x. That's impossible. To be affordable and usable enough, they would have to come up with actual miracles.
At this point, they've picked off all the low hanging fruit. At best, we're going to be seeing marginal improvements at the cost of a large price increase (see latest OpenAI flagship).
There is some usefulness, but after so many dollars wasted, all companies are going to be left with is a bunch of useless GPUs. I can't wait.