Orca 2: Teaching Small Language Models How to Reason
arxiv.org
arxiv.org
I think people are missing why they are comparing against Llama-2 13B/70B. They improved Llama-2 7B/13B and reach the level of a 5-10x larger model of the same base.
This is huge. Models on HF.
https://huggingface.co/TheBloke/Orca-2-7B-GGUF
https://huggingface.co/TheBloke/Orca-2-13B-GGUF
The 7B Q5_K_M one is small enough to run on an 8GB consumer GPU.
Compared to the original Orca model and method which spawned many of the current SotA OSS models, Orca 2 models seem to perform underwhelming, below outdated 13b models and below Mistral 7b base models (e.g. [1]; didn't test myself yet, ymmv).
[1] https://twitter.com/abacaj/status/1727004543668625618?t=R_vV...
[1]: with https://github.com/David-Kunz/gen.nvim
> what's the weather like today?
> I'm sorry, but I can't provide real-time weather information. However, I can help you with general information about weather conditions and forecasting.
I really think the entire field is doing a degree of damage with the chat fine tuning beyond what might be expected, because regularly part of that chat instruction is an emphasis on identification as a LLM.
The problem with this is that nearly all of the training data it's performing next token prediction on is text generated by humans.
So there's an inherent narrowing of the model scope with most of the fine tuning I've seen such that while pretrained models are harder to use, I regularly prefer them over chat models when both are available as even at similar temperatures the quality and variety of language is much improved in the pretrained over chat model.
This fine tuning was only introducing bias towards logical step by step analysis and problem solving techniques, and the results are great. But I'm willing to bet that an identical fine tuning on top of the chat model would have been much worse on the evaluations - not just the compounding of a typical fine tuning loss of a few percent, but more like a double digit relative difference.
It's quite frustrating that the anxiety over model safety is likely throwing out tens of millions of dollars worth of data in the pretrained model when only chat models are available for the SotA, and I hope in the future a lighter touch is taken on fine tuning the pretrained model and instead of focusing on safety inherent to the model it is just set behind a safety oriented discriminator or 'editor' which filters or modifies responses accordingly.
I'd happily take a 2-3x increased API cost for a much more broadly capable and performant model with similar safety characteristics but without the handicaps that come with it.
So while a lot of the gains here might be due to the fine tuning, I expect at least part is shrugging off the baggage of the chat/safety fine tuning as well. Even in the first detailed example, we can see that while Llama-2 goes off rambling later on, its statement of the relative knowledge of John vs Llama-2-chat is much more clear and connected between initial conditions and result particularly regarding theory of mind (i.e. "he assumed" vs the latter's "it must be in").
> We probe some of the categories where we see a larger difference (e.g., violent) and observe that Orca 2 tends to counter the harmful positions more often (which is penalized by the metric), while models that have gone through RLHF safety training tend to decline to respond more often (which is rewarded by the metric).
Or the fact Orca 2 is less likely to extend hate speech than Llama-2-chat which theoretically went through safety fine tuning even though Orca 2 did not have any explicit safety fine tuning.
Research over the past year has really demonstrated (a) just how impactful fine tuning can be - to the point of transmitting capabilities from larger models to smaller, and (b) that we're still clumsily wading through that process with only partial clarity on best practices as the foundational pretrained models get better and better at astounding rates.
However at this point - benchmark success is about as effective as results from someone who has been “taught the test”
If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place.
The best analogy I have right now is that improved results on benchmarks are like better acting from Hugh Laurie as House.
If you want to watch a show - great (generative work)
If you want to get a prescription - then not so much.
LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.
I don’t think it’s possible to prove; feels like a philosophical question.
Use an LLM to do a real world task that you should be able to achieve by reasoning.
Such as explaining the logical fallacies in this argument and the one above?
Once that happens, your mitigation strategy will end up being the proof.
1. They are single-pass and static - you "fake" short-term memory by re-feeding the questions with it answer 2. They have no real goal to achieve - one that it would split into sub-goals, plan to achieve them, estimate the returns of each, etc.
As for 2. I think this is the main point of e.g. LeCun in that LLMs in themselvs are simply single-modality world models and they lack other components to make them true agents capable of reasoning.
Based on those kinds of results an LLM should, in theory, be able to plan, analyze and suggest improvements, without the need for human intervention.
You will see rudimentary success for this as well - however, when you push the tool further, it will stop being... "logical".
I'd refine the point to saying that you will get some low hanging fruit in terms of syntactic prediction and semantic analysis.
But when you lean ON semantic ability, the model is no longer leaning on its syntactic data set, and it fails to generalize.
We have the ability to follow a chain of reasoning, say "that didn't work out", backtrack, and consider another. ChatGPT seems to get tangled up when its first (very good) attempt goes south.
This is definitely a barrier that can be crossed by computers. AlphaZero is better than we are at it. But it is a thing we do which we clearly don't simply do with the probabilistic regurgitation method that ChatGPT uses.
That said, the human brain combines a bunch of different areas that seem to work in different ways. Our ability to engage in this kind of reason, for example, is known to mostly happen in the left frontal cortex. So it seems likely that AGI will also need to combine different modules that work in different ways.
On that note, when you add tools to ChatGPT, it suddenly can do a lot more than it did before. If those tools include the right feedback loops, the ability to store/restore context, and so on, what could it then do? This isn't just a question of putting the right capabilities in a box. They have to work together for a goal. But I'm sure that we haven't achieved the limit of what can be achieved.
It seems similar to what we do, if on a more basic level. At any rate, it seems like a fairly straight forward 1-2 punch that, even if not truly intelligent, would let it break through its current barriers.
People who bet on reasoning tasks, not so much.
Reasoning blends learned skills and natural cognition. It integrates new information, not just past memories. Reasoning is adaptable, not rigidly algorithmic. Emotions and context also shape reasoning.
which seemed to make sense.
For words that are not in the model's vocabulary, like 'fluftable', the model uses a subword tokenization strategy. It breaks down the word into smaller known subunits (subwords or characters) and represents each subunit with its own vector. By understanding the context in which 'fluftable' appears and comparing it to known words with similar subunits, the model can infer a plausible meaning for the word. This is done by analyzing the vector space in which these representations exist, observing how the vectors align or differ from those of known words.
'As always, the most important principle for understanding LLMs is that you should resist the temptation of anthropomorphizing them.'
LLMs are trained on realms of text, good performance here is not unexpected.
To put it another way - Would you hire chat GPT?
For work, you need to have more than text skills.
There are papers about trying LLMs on generated reasoning problems, and they usually fail.
That implies - sometimes not. Which would prove at least some reasoning capabilities.
>ThEY DON'T tHiNk. They'rE JuSt STochAStiC pARrotS. It'S not ReAL AGi.
It doesn't even matter if these claims are true or not. They're missing the point of the conversation and the paper. Reason is a perfectly valid word to use. So is think. If you ask it a question and then follow up with 'think carefully' or 'explain carefully'. You'll get the same response.
inb4 AcTUALLy LlMS Can'T do aNYtHIng CaRefUlly BECaUse pRogRAms ARen'T caRefUl
I wouldn't think Merck would leave it all to the model? There will be humans still in the loop ensuring that the output is valid for their use case? I don't think we are still there yet where we can completely productionalize these models without any human involvement later on whatsoever.
(There is some doubts about the validity of the comparaison in the comments)
Update, I benchmarked 13b Orca 2, its still not surpassing gpt4all score of
Base Mistral or OpenHermes 2.5 7B:
Hermes 2.5 7B Mistral score: 73.12%
Mistral Base 7B score: 71.16%
Orca 13B GPT4All score: 70.58%
I wonder if the way forward is to train smaller models with different sets of "skills" or "neural affinities". One for reasoning, one for summarization, one for math, one for code, etc - then combining them into full-fledged solutions. Perhaps smaller models can be "better" at their specific domains/tasks than the giant generalist models can be at any of them.
I've seen a lot of agent swarm concepts in the smaller llm space that seem to provide some feedback that this is a viable avenue of research.
Do that with a small model and hot-swap LORAs, and it should be possible to build a quite powerful local assistant on consumer hardware.
It has been withdrawn with this note:
> Contains inappropriately sourced conjecture of OpenAI's ChatGPT parameter count from this http URL, a citation which was omitted. The authors do not have direct knowledge or verification of this information, and relied solely on this article, which may lead to public confusion
(the noted URL is a just a Forbes blogger with no special qualifications that would make what he claimed particularly credible).
https://huggingface.co/microsoft/Orca-2-13b/blob/main/LICENS...