Hacker News Reader

Comment by devsda | Hacker News Reader

I don't understand LLMs enough to know if this is a silly question or not.

Is it possible to build domain specific smaller models and merge/combine them at query/run time to give better response or performance instead of one large all knowing model that learns everything ?

View on HN

I think that's the intuition behind MoE (Mixture of Experts). Train separate subnets for different tasks, train a router that selects which subnets to activate at inference time. Mixtral is a current open model which I believe implements this.

No. MoE tends to change expert every other word. There’s a bit of pattern (like a lot of punctuation to one expert) but it’s not clear what. Nobody understands how or why the router chooses the expert. It’s so early.

> Nobody understands how or why the router chooses the expert. It’s so early.

Nobody understand how LLM works either. Is LLM as "early" as MoE ?

LLMs are really well understood, what do you mean? You can see the precise activations and token probabilities for every next token. You can abliterate the network however you'd like to suppress or excite concepts of your choosing.

There's various layers of understanding.

If you will excuse analogy and anthropomorphism, the human analogy of what we do and don't understand about LLMs is, I think, that we understand quantum mechanics, cell chemistry, and overall connectivity (perceptrons, activation functions, and architecture) and group psychology (general dynamics of the output), but not specifically how some belief is stored (in both humans and LLMs).

Mathematically speaking LLMs have very precise formulation and can be seen as F(context, X0, X1, ..., XP) = next_token. What science behind the LLMs is still lacking is how all these parameters are correlated one to each other and why one set of values is giving a better prediction than the other set of values. Right now, we arrive to these values through experimental approach, that is, through trainings.

i think the younger generation who came up post deep learning, has a very very low bar for “understanding” because they never knew a world where SotA models worked in a way that made sense.

> MoE tends to change expert every other word

Any citation on this one?

It's covered in the original Mistral "Mixtral of Experts" paper [0].

0. https://arxiv.org/abs/2401.04088

I believe it's actually a per token routing, not a "every few words"

It's mechanically capable of per-token routing, but the routing tends to be stable across more than one token. It's weird.

It's got nothing to do with words, and many MoEs route to multiple experts per token (the well known Mixtral variants for example activates 2 experts per token).

Weirdly it does have to do with words, but not intentionally. Mechanically the routing is per-token, but the routing is frequently stable across a word as an emergent property. At least, that's how I read the mixtral paper.

Yep.

Also note that I’m ELI5 so saying word is fine.

This is not how MoEs work at all. They are all trained together, often you have multiple experts activated for a single token. They are not domain specific in any way that is understandable by humans.

You might want to look into "task arithmetic" which aims at combining task-specific models post-training. For example:

https://proceedings.neurips.cc/paper_files/paper/2023/file/d...

It's possible, the question is how to choose which submodel will be used for a given query.

You can use a specific LLM, or a general larger LLM to do this routing.

Also, some work suggest using smaller llms to generate multiple responses and use a stronger and larger model to rank the responses (which is much more efficient than generating them)

Taking a further step back from LLM’s, this is called portfolio / ensemble techniques in the literature.

A common practice in more formal domains is to have a portfolio of solvers and race them, allowing for the first (provably correct) solver to “win”

In less formal domains, adding/removing nodes/trees in an online manner is part of the deployment process for random forests.

This is called speculative decoding

No, speculative decoding is when you use a smaller draft model to propose tokens and then use the larger target model to verify the proposals. It has got nothing to do with domain specialization.

Reply on news.ycombinator.com