50% on HumanEval with just 1.3B model

For reference, this beat GPT-3.5 which scores 47%, but not GPT-4 which scored a massive 67%.

Beating out GPT-3.5 at any task with such a small model is very cool to me.

How much longer until these dumb virtual assistants (Siri, Google, Alexa) get replaced with on-device LLMs? We’ve gotta be getting close. These small, optimized models are catching up quickly in so many domains.

Dupe: "Textbooks are all you need" https://news.ycombinator.com/item?id=36413768

I wonder if learning to train Generative AIs will teach us anything about teaching humans? I mean other than the use of AI tutors. Can we determine the usefulness of text by how well it trains AI?

What is B in this context? Billion?

Yes, nB is n-billion number of parameters

Yes, 10^9.

What is HumanEval?