Hacker News Reader

Comment by enum | Hacker News Reader

StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages.

Also available in HuggingChat: https://huggingface.co/chat

A similar model from Replit was released this week: https://huggingface.co/replit/replit-code-v1-3b

The performance of StarCoder seems superior on HumanEval pass@1:

• replit-code-v1-3b: 21.9%

• replit-finetuned-v1-3b: 30.5%

• StarCoder: 33.6%

• StarCoder prompted: 40.8%

Also note that StarCoder itself is open (but not replit-finetuned).

The quality delta is so huge that it's in a different league.

Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.

If so, it’s a shame this post is getting comparatively little attention. Hopefully the quality will be enough for it to make a big splash somewhere.

I'm assuming higher is better? How does this compare to GPT-3.5/4?

I think it's still significantly behind GPT 3.5/4, both of which can get 67% on HumanEval, and 88% with Reflexion

Keep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.

Aren't GPT-3 etc base LM and ChatGPT the instruction tuned? Or am I wrong?

code-davinci-002 is a base LM, and the other 3.5 models (text-davinci-{002,003}, gpt-3.5-turbo, and ChatGPT) use instruction tuning and/or RLHF. Source: https://platform.openai.com/docs/model-index-for-researchers