Hacker News Reader

Comment by swyx | Hacker News Reader

i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code.

GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last.

Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestral with additional pretraining (it seems like we don't know how much more tokens are on this one, but CodeLLama was 500B-1T extra tokens of code).

Zuck has mentioned recently that he doesnt see coding ability as important for his usecases, whereas obviously OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI.

View on HN

>Zuck has mentioned recently

That's a really surprising thing to hear, where did you see that? The only quote I've seen is this one:

>“One hypothesis was that coding isn’t that important because it’s not like a lot of people are going to ask coding questions in WhatsApp,” he says. “It turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical structure of knowledge, and just generally have more of an intuitive sense of logic.”

https://www.theverge.com/2024/1/18/24042354/mark-zuckerberg-...

Make Sense, they want better interaction whit users for Whatsapp, Instagram and Facebook marketers, content creation and moderation,and their glasses(ai /ar) I don't see in that context why the should push more effort into llm coding, is sad anyways

He mentioned it on the Dwarkesh podcast: https://www.youtube.com/watch?v=bc6uFV9CJGg

I watched this podcast and i also remember zuck saying it is important

> OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI.

And researchers from Google Deepmind, University of Wisconsin-Madison and Laboratoire de l’Informatique du Parallélisme, University of Lyon, actually publish some of their results in that direction [1,2].

[1]: https://deepmind.google/discover/blog/funsearch-making-new-d...

[2]: https://www.nature.com/articles/s41586-023-06924-6

The OpenAI philosophy is that adding modes improves everything. Sure, it’s astronomically expensive, but I tend to think they’re on to something.

> there's no CodeGPT, its just GPT4

Codex[1] is OpenAI's CodeGPT. It's what powers GitHub Copilot and it is very good but not publicly accessible. Maybe they don't want something else to outcompete Copilot.

[1] https://openai.com/index/openai-codex/

> Zuck

No, if anything he said Meta realized coding abilities make the model overall better, so they focused on those more than before.

I thought that was the idea, open source small specific models that most people can run vs general purpose ones that require a massive amount of GPUs

Reply on news.ycombinator.com