BigCode Project Releases StarCoder: A 15B Code LLM | Hacker News Reader

BigCode Project Releases StarCoder: A 15B Code LLM

huggingface.co

185 pointsenum37 comments

Open articleView on HN

StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages.

Release thread: https://twitter.com/BigCodeProject/status/165417494197606811...

Model: https://huggingface.co/bigcode/starcoder

Paper: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k...

Also available in HuggingChat: https://huggingface.co/chat

A similar model from Replit was released this week: https://huggingface.co/replit/replit-code-v1-3b

The performance of StarCoder seems superior on HumanEval pass@1:

• replit-code-v1-3b: 21.9%

• replit-finetuned-v1-3b: 30.5%

• StarCoder: 33.6%

• StarCoder prompted: 40.8%

Also note that StarCoder itself is open (but not replit-finetuned).

The quality delta is so huge that it's in a different league.

Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.

If so, it’s a shame this post is getting comparatively little attention. Hopefully the quality will be enough for it to make a big splash somewhere.

I'm assuming higher is better? How does this compare to GPT-3.5/4?

I think it's still significantly behind GPT 3.5/4, both of which can get 67% on HumanEval, and 88% with Reflexion

Keep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.

Aren't GPT-3 etc base LM and ChatGPT the instruction tuned? Or am I wrong?

code-davinci-002 is a base LM, and the other 3.5 models (text-davinci-{002,003}, gpt-3.5-turbo, and ChatGPT) use instruction tuning and/or RLHF. Source: https://platform.openai.com/docs/model-index-for-researchers

https://newatlas.com/technology/gpt-4-reflexion/

License text: https://huggingface.co/spaces/bigcode/bigcode-model-license-.... Attachment A at the very end has the usage restrictions.

Very interesting read.

OK for some commercial uses but I wonder how many other models have restriction lists like this

Some parts of ethical licensing seem to finally pick up momentum, seems like they are using a standard license: https://www.licenses.ai/blog/2022/8/26/bigscience-open-rail-...

To me it would be even more responsible if they would define a default output license compatible to training input of the model. I guess they could simply refer to the input data set in there to fulfill attribution clauses of most permissive licenses. Or extract some really long list of copyright notices from the sources.

Well rather than "ethical" licensing, I'd argue this is "moral" licensing, and as usual, morals differ. For example:

> (m) To provide medical advice or medical results interpretation that is intended to be a substitute for professional medical advice, diagnosis, or treatment;

This seems particularly overly broad -- there is a lot of benefit for people with no access to a doctor of any kind that is left on the ground by excluding that use case.

The problem of bad advice is hard, but it's one that will likely become increasingly solved as the tech evolves, and this seems to exclude this model from that technological progression.

Does this work for second degree usecases? Since this a codegen model, can I output code that does something like fast-edge detection, put it under MIT, and then use it for medical analysis?

Who knows… until any of this gets tested in a court of law we can only guess (with legal professionals having the best guesses)

So another model that isn't actually open source.

This basically has the same restrictions as LLAMA and stable-diffusion. Replit is licensed under cc-by-sa-4.0 which is truely permissive.

Paper: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k...

Evaluation results are impressive!

https://huggingface.co/spaces/bigcode/bigcode-playground

Can someone explain to me the "Model Formats" section at the end of that page. It makes sense to me as a description of how to format the training data, but it also says "Use these templates to explore the model’s capacities"

How would you make use of this prefix format in a prompt?

  <reponame>REPONAME<filename>FILENAME<gh_stars>STARS
  code<|endoftext|>

You may (but do not have to) use <reponame>, <filename>, etc. as special tokens to prompt the model with extra metadata. These help you use the model go beyond just code completion.

Page 30 of the TR has a few examples:

https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k...

Note that they were produced with StarCoderBase. E.g., here is one:

Input:

    <commit_before>def fibonacci(n):<commit_msg>add type hints to function<commit_after>def

Output:

    fibonacci(n: int) -> int:

(If you try this in the playground, dial down the repetition penalty to say 1.0 in Advanced Settings.)

StarCoder is not just a code predictor, it is an assistant. It doesn’t just predict code; it can also help you review code and solve issues using metadata, thanks to being trained with special tokens.

The <reponame> token specifies the name of the repository, and the same goes for the filename. A high gh_stars count can make it mimick popular open-source libraries. Admittedly, there is a bit of information that can be retrieved from those tokens, but not as much as the others.

The more interesting ones are the commit tokens, which you can use to ask it as an assistant to implement a commit whose high-level description you give in plain English, and the issues token, to make it solve an issue.

The Fill-in-the-middle tokens are most useful for autocompletion: it will allow it to gain information from the code that is present after your cursor, instead of just the code before your cursor.

Of course, in practice, those tokens are meant for code editor plugin writers. Normal users won’t know about them.

Is it possible to load the model for a particular language. Thus minimising the required resources.

The org page has a link to https://huggingface.co/bigcode/starencoder "Encoder model trained on TheStack." which is currently a 404

Just curious what that is vs the main https://huggingface.co/bigcode/starcoder model

This was used for PII detection and will be public soon.

It could have other uses, but its not what you want code generation. For code generation, use StarCoder or StarCoderBase.

I was wondering about maybe embeddings for code search

I'm excited to see all the model's implementations that are yet to come. Refact.ai seems to be working on code transformation tools and chat inside Jetbrains and VSCode powered by StarCoder https://refact.ai/blog/2023/self-hosted-15b-code-model/

The training cluster is super impressive, any ideas on how to fine tune StarBase for a smaller language than python on something more attainable?

Suggested fine-tuning code is here:

https://github.com/bigcode-project/starcoder

Thank you!

i was not able to find the 80 programing languages it supports using my phone. you would think this would be front and center somewhere.

There is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLMs.

But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.

You are correct that there is no evaluation of level of support, and it is hard to get evaluation set on 86 languages. On the other hand, we can try to extrapolate from what we have and try to guess in which languages we'd have reasonable performance. Note that out of 19 languages it was evaluated on only 17 were "officially" in the training set (language detection is not perfect and there may be some data for languages not included in training set) and it work reasonably well on the remaining two (Swift and D).

I might have missed it, what are the minimum hardware requirements needed to run this?

You can just about load it on a 32GB GPU in 16bit mode.

Quantized versions here:

https://huggingface.co/mayank31398/starcoder-GPTQ

they will be benchmarked on humaneval and released soon—maybe tomorrow?

So many things are 404 on this announcement. The demo looks really amazing though, I’m pretty excited to try this out locally.

Reply on news.ycombinator.com