Orca: Progressive Learning from Complex Explanation Traces of GPT-4
arxiv.org
arxiv.org
I’m surprised this hasn’t received more comments. Orca is an important result. An 11B-parameter model has reached near-parity with a 175B-parameter model on reasoning tasks. You don’t have to squint hard to see where this work is headed.
The idea (and model) is very exciting. I figured efficiency was going to be the next area of breakthrough, since monolith LLMs seem to be VRAM-limited these days.
I hope this is a sign of things to come and ML becomes more democratized!