Hacker News Reader

Comment by mousetree | Hacker News Reader

How does this compare to Github Copilot? It's not shown in their comparison

Not sure how much current Copilot varies from the original Codex, but another set of benchmarks here: https://paperswithcode.com/sota/code-generation-on-humaneval

It is fast all right, but the quality is not there. I asked it to implement OAuth with Stytch and Ktor and it made everything up. I pointed out the correct name for the package and asked if it really knew the SDK, and it apologized and repeated the same made up code after merely changing the name of the package.

This is actually why we (Stytch) haven't rolled out any of these "chatbots for code".

We have a big list of example questions we get from devs trying us out and we've tested several home grown and third party providers and thus far haven't seen anything good enough that we'd put into production.

Thanks for testing this out for us! I'll cross it off our list :)

It's miles better.

In fact I stopped using expensive GPT-4

Codestral just works, its quick, output is accurate its kinda scary.

Knowing the training data GH has I doubt it's comparable, then again I don't have the benchmarks

After typing this I tried the live chat out and it honestly seems a lot more promising than current GH Copilot, very nice!

Are you saying GH has more than Codestral and therefore GH has a better model? Or that Codestral would be better because Codestral is not littered with "bad" code?

Bad code is obviously very subjective, but I would wager that GH places a much higher value on feedback mechanisms like stars, issues, PRs, velocity, etc. Their ubiquity likely allows them to automatically cherry-pick less "bad code."

Nothing prevents Mistral do the same if they want to. Issues and and PRs are public information, exposed by APIs, and not that much rate limited.

Copilot primarily uses GPT-3.5, which is outclassed by Llama3-70B. And this model claims to be slightly better than Llama3-70B.

Edit: For those who don't believe me, https://github.com/microsoft/vscode-copilot-release/issues/6.... Gpt-4 for chat, 3.5 for code.

GitHub Copilot uses GPT-3.5?

I was under the impression it was a custom codex model with a surrogate local model as per https://github.blog/2023-02-14-github-copilot-now-has-a-bett...

When did this change?

When it first launched it, I too didn't know they had changed the model from the original codex which came similar time as gpt-3.5

> Gpt-4 for chat, 3.5 for code

That thread is comparing sidebar chat to inline chat. Doesn't discuss code completions afaict.

Reply on news.ycombinator.com