Hacker News Reader

There is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLMs.

But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.

View on HN

You are correct that there is no evaluation of level of support, and it is hard to get evaluation set on 86 languages. On the other hand, we can try to extrapolate from what we have and try to guess in which languages we'd have reasonable performance. Note that out of 19 languages it was evaluated on only 17 were "officially" in the training set (language detection is not perfect and there may be some data for languages not included in training set) and it work reasonably well on the remaining two (Swift and D).

Reply on news.ycombinator.com