Hacker News Reader

Comment by wewewedxfgdf | Hacker News Reader

Can someone explain to me please why Meta doesn't create subject specific versions of their LLMs such as one that knows only about computer programming, computers, hardware software.

I would have imagined such a thing would be smaller and thus run on smaller configurations.

But since I am only a layman maybe someone can tell me why this isn't the case?

View on HN

Generally, all that non-tech content still helps the model “to learn”.

Also, the software you’re working on will generally in some way have a real-world domain - without knowing it the AI all likely be a less effective assistant. Design conversations with it would likely be pretty non-fun, too.

Finally, the “bitter lesson” article[0] from a couple years ago is I think somewhat applicable too.

[0]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

One of the weirdest and most interesting parts of LLMs is that they grow more effective the more languages and disciplines they are trained in. It turns out training LLMs on code instead of just prose boosted their intelligence and reasoning capabilities by huge amounts.

Source? Sounds interesting

There are a lot of papers that stumble upon this while working with LLMs and its mentioned off hand in the conclusions. If you want a paper that explicitly tested this here you go - https://aclanthology.org/2022.coling-1.437.pdf

To add on to the sibling. Specialized models, including fine tuned ones, continually have their lunch eaten by general models within 3-6 months. This time round is mixture of experts that’ll do it, next year it’ll be something else. Tuned models are expensive to produce and are benchmark kings but less do less well in the real world qualitative experience. The juice just ain’t worth the squeeze most of the time.

Meta does have some specialized models though, llamaguard was released for llama 2 and 3.

> Tuned models are expensive to produce

The expensive part is building the dataset, training itself isn't too expensive (you can even fine-tune small models on free colab instances!), and when you have your dataset, you can just fine tune the next generalist model as soon as it's released and you're good to go ago.

Other companies have done this (see Qwen Coder). It doesn't scale past a few disciplines like math and code though, and using mixtures of experts give you most of the same benefits.

Unlike what their names imply, MoE don't have a concept of domain expert at all.

It's just a fancy name for sparse evaluation of the total network to save compute and memory bandwidth.

Reply on news.ycombinator.com