Hacker News Reader

Comment by docheinestages | Hacker News Reader

Given the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture.

I think it's very valuable to have a smaller model that doesn't have any domain knowledge or facts built into its weights, but given the right context, could accurately reason about what to do and use the right tools.

I'm aware of colibri [1], but so far I've only seen extremely slow performance.

[1] https://github.com/JustVugg/colibri

View on HN

"I'd like a car that goes 300mph and gets 100mpg while doing it. I'm aware of a car that gets 100mpg but it is extremely slow."

You are describing fundamental tradeoffs. Getting more performance relative to model size and training token amount is what all of the labs are solving.

Labs are focusing on creating models, small or large, that perform well on various benchmarks, including general knowledge, domain-specific expertise, and agentic capabilities.

Asking for such a model while wanting to be small and fast would align with what you're describing, which I believe is different from what I'm pointing to.

The model I'm describing sacrifices domain knowledge and expertise for agentic reasoning and tool-calling capabilities at a reasonable speed.

Think of Cactus Compute's Needle [1].

[1] https://cactuscompute.com/blog/needle

The core intuition here is probably that you can not separate reasoning and domain knowledge.

How are you going to use tools, if you don't have the context to use them?

Imaging if you could only think in terms of lambda calculus, and was asked to check the weather to let me know if the weekend is good for a hike.

I think the idea here would be to not use your super smart but specialized model to check the weather. It's not obvious that it's impossible to (eg) remove most of its biology knowledge, without removing much of its ability to develop software (for example). If you're developing biology software, then don't use that particular compressed model.

(If you're claiming that it is impossible, and you have references you can share, then I'm honestly interested.)

I don't think that is true.

I think more of the capabilities comes from the cross domains knowledge.

Anyways, software is also designed for a domain. The reason why it is so adept at making a fitness tracker is likely because it knows about biology.

Sure, that's a plausible theory but I haven't seen that anyone's proved it.

An alternate theory is that models need lots of input knowledge to learn complex reasoning, but don't need so much at inference time. An example is arithmetic. Early in their training, models do arithmetic (poorly) by pattern-matching on memorized examples. Eventually they grok arithmetic and stop pattern matching, and then they don't use the examples anymore.

The fitness tracker doesn't take much knowledge of biology. Large models have quite a lot of biology knowledge that most people will never use. Same goes for lots of other topics. For basic knowledge that easily fits in context it could search the internet, or a local collection of introductory textbooks.

There’s another way besides distillation that’s way cheaper: You can have the big model build prescriptive skills that the small model follows.

Take the “train” portion of tasks on some benchmark, have K3 complete it, and then output detailed descriptions of tools used and why, then run the validation tasks with some small model that has access to the skills.

Yes. Using a harness with a strong model to create lots of utilities and tools for yourself is effectively the same thing.

Isn’t that distillation ?

No. Distillation trains on a teach model's logits or output tokens.

How do you have teacher model output anything without it being output tokens (or embeddings / intermediate logits / activations)?

Why not generate an artificial dataset using commercial APIs and then finetune a small model on this data?

I’ve had success adapting even a 7B model for single-domain tasks that way, including reasoning and tool calling.

You can use an open model. The point is just to outsource the inference, so you don’t have to deal with running the larger model yourself.

This is probably one of the ways to achieve this. I see a plethora of such fine-tunes on HuggingFace [1], but they're either not much different than the base model or they're outright benchmaxxing.

[1] https://huggingface.co/models

> smaller model that doesn't have any domain knowledge or facts built into its weights

I’m not sure it works this way. Language modelling itself is a “domain”, and if it didn’t have a grasp language it wouldn’t be able to do anything else.

I also think a “reasoning engine” that had to reason through everything from first principals would likely be extremely inefficient.

It’s good went models have domain knowledge and expertise - and all of their reasoning flows downstream of that

Reply on news.ycombinator.com