It could be that your problem was too simple to justify the use of Deep Think.
But yes, Google should have figured that out and used a less expensive mode of reasoning.
But yes, Google should have figured that out and used a less expensive mode of reasoning.
This is why model pickers persist despite no one liking them.
Once you've given the model your prompt and are reading the first output token for classification, you've already paid most of the cost of just prompting it directly.
That said, there could definitely be exceptions for short prompts where output costs dominate input costs. But these aren't usually the interesting use cases.
Source? Afaik this is incorrect.
Caching does help the situation, but you always at least pay the initial cache write. And prompts need to be structured carefully to be cacheable. It’s not a free lunch.
I don't disagree that there are hard problems which use short prompts, like math homework problems etc., but they mostly aren't what I would categorize as "real work". But of course I can only speak to my own experience /shrug.
And a question to the knowledgeable: does a simple/stupid question cost more in terms of resources then a complex problem? in terms of power consumption.
Even proof mining and the Harrop formula have to exclude disjunction and existential quantification to stay away from intuitionist math.
IID in PAC/ML implies PEM which is also intentionally existential quantification.
This is the most gentle introduction I know of, but remember LLMs are fundamentally set shattering, and produce disjoint sets also.
We are just at reactive model based systems now, much work is needed to even approach this if it ever is even possible.
[0] https://www.cmu.edu/dietrich/philosophy/docs/tech-reports/99...
I see this semi-regularly: futile attempts at handwaving away the obvious intelligence by some formal argument that is either irrelevant or inapplicable. Everything from thermodynamics — which applies to human brains too — to information theory.
Grey-bearded academics clinging to anything that might float to rescue their investment into ineffective approaches.
PS: This argument seems to be that LLMs “can’t think ahead” when all evidence is that they clearly can! I don’t know exactly what words I’ll be typing into this comment textbox seconds or minutes from now but I can — hopefully obviously — think intelligent thoughts and plan ahead.
PPS: The em-dashes were inserted automatically by my iPhone, not a chat bot. I assure you that I am a mostly human person.
Who do you think is going to be successful, those who realize the limitations and strength of a system and leverage them, or those who are complacent, with a unwarranted self-satisfaction accompanied by unawareness of actual risks or deficiencies of a particular system?
IMHO they are always going to be too complex to know everything about a models of this size, but there are areas we do know their limits or the limits of computation in general.
But feel free to stay on your high horse and call people names and see how well that works out for you.
"Here I am, brain the size of a planet, and they ask me to ..."
Well, I don't think it's easy or even generally possible to recognize a problem complexity. Imagine you ask for a solution for a simple expressed statement like find an n > 2 where z^n = x^n + y^n. The answer you will receive will be based on a trained model with this well known problem but if it's not in the model it could be impossible to measure its complexity.