Hacker News Reader

Comment by giancarlostoro | Hacker News Reader

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory.

Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. I wouldn't be surprised if frontier AI could be drastically cheaper if they just finetune and optimize their models to not consume all available RAM to only access less than 10% of the models knowledge.

View on HN

I don't care who King Charles is every single time

that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

This makes me think (by full ignorance) how quantum computing can be useful in this field of things reach a stable-ish point

they will not. quantum computers as understood today will never run an LLM. read = destroy. and no cloning. wanna reload the entire model for each token? preparing quantum states is slow by any standard.

it's just not something you would ever use a quantum computer for

Seems to me like there's a conflict of incentives in the current era of market capture.

Not to say the cost-cutting wouldn't be valuable; today's race is predominantly about the model's reasoning capacity or "how hard of a math problem can the model solve".

It'll be a nice day when research-oriented human capital gets redirected to things that benefit us layfolks's pockets more directly

The human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up.

So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

LLMs can do that. The one that's built in to google search is relatively stupid, but it just uses web pages to fill in the gaps.

The downside is it is incredibly easy to be tricked by a single false bit of info.

I guess that's why is an active research area, and an interdisciplinary one, what is intelligence? Is what have you practiced a ton of times? Is what you have purposely and efficiently practice? What is the implication of "purposely" doing something and how much memory is involved into it? If memory is involved how much of it is relevant? What is problem solving or wisdom? creativity? How much diversity in your memory do you need for creativity?

So what should you cull and how much? There's already techniques in CNNs to trim unused or less active neuron paths to reduce a model's size, but how do you (and how much) do it in a general LLM? A product that will be used without supervision from a child to an 80yo elder?

> So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

MoE (mixture of experts)?

I read a lot in middle school and can't recall any of them. What books did you read in college? What books did you read in high school?Maybe you have photographic memory.

cant we have specialised models during onboarding i doubt a developer needs to know who king charles is?

We can and we do https://en.wikipedia.org/wiki/Mixture_of_experts and even then - maybe not every developer needs python, maybe some do need C++ and King Charles..

Claude, create a webpage that provides family tree diagrams of all European royal families, clicking on each name expands the element to include any notable events from that person's life.

I've been dealing with it on the post-training (runtime) side with a large codebase that contains many moving parts, many rules and requirements. Putting all of that into the context has already gone far past 1MB, so it's untenable.

Now I have all of that information stored in a pointers-to-resources fashion, where layers of small "directories" of triggers-to-information point the agent gradually towards deeper, more esoteric knowledge the more specific its needs become when tackling a goal.

distill. start with a general model, then create a smaller model trained only on relevant coding examples.

I have some ideas on how that specifically can be solved, but I've taken a stance to never give OpenAI or Anthropic any of my ideas for free. I am the most surprised that Google seems to be trailing behind them. I'm not sure if they're even taking this seriously anymore. I do appreciate the open models they do release on the other hand, I hope they never stop. I wish Microsoft would do more with Phi and similar.

> I have some ideas on how that specifically can be solved

Write them in the margin of a book....

"It is impossible for a cube to be a sum of two cubes, a fourth power to be a sum of two fourth powers, or in general for any number that is a power greater than the second to be the sum of two like powers. I have discovered a truly remarkable proof, but this margin is too small to contain it."

Email it to my gmail account. It's secure from prying eyes :)

> I don't care who King Charles is every single time

There is just a stupendous amount of everything packed into a 35B size or larger model. For instance Qwen 3.6 35B A3B (Q8) can do a fairly decent job translating English to Arabic, but it can also generate python code with a reasonable layout and commenting.

I try to remember that as a mental model, an epub copy of a truly gargantuan sized 1000+ page novel such as the unabridged/2nd revision of Steven King's The Stand is about 800KB, and we're talking about a GGUF file that's 37GB in size or something like that.

> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time.

That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.

I know very little about this but it seems like the kind of thing that can and eventually will be solved computationally, not by people figuring out what a parameter or group of parameters represent

It's not a compute problem. It's a knowledge problem. Even if you can process each parameter individually and re-link the model layers, you need enough information to know what each parameter is for which is necessarily more memory than the weights themselves. You can use the weights to know whether each parameter is useful for a given prompt, but that operation is a strict superset of just generating the answer. By the time you know which parameters are useful, you've already done all the work of generating your output tokens and the effort is pointless.

Would an extension to `madvise` to say: "page this region in/out as a whole" help here? Engine could define memory ranges representing each expert and leave paging to the OS (tho' "paging" at this point becomes more similar to swapping in granularity...).

Experts are chosen per token

> I don't care who King Charles is every single time

Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

> Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

I think this can be achieved already. Take a base model and train only on source code. In fact, the very early Granite models from IBM were like that though it didn't support reasoning which limited its performance.

You can do it too. I don't know how much it will cost to train on just source code repos. $10K in total? Not sure.

I’m a bit skeptical. Without instructional materials from textbooks, programming language reference manuals and guides, as well as general knowledge about logic and discrete mathematics, I doubt a model could work very well.

Yes. Model would be limited in its performance, but it would perform well in the limited domain because LLM interpolate from training data. They don't think like humans.

We might think that knowledge from logc and discrete math would spill over to coding. Unfortunately, it doesn't seem to work like that. Even 1T parameter LLM fail on tasks if there are no variants of it in the training data.

I'm skeptical that the "logical ability" is much more then the elided data. Obviously some things get fully memorized and other things don't, but I don't think there's anything like functional circuits.

Isn’t this essentially what MoE partially solves with varying levels of accuracy?

Sadly no. Despite the name, the experts are not routed per concept or topic but per token. So for the same sentence you might activate multiple experts for different tokens. What it solves is the distributed training and inference problem. As long as each expert fits a single gpu, coordinating the model evaluation is much easier and it is faster. It does not buy as much for running on a single device though still less costly than a dense version.

Apple’s new Foundation model for the 27 OS releases does some interesting things in exactly this area: https://machinelearning.apple.com/research/introducing-third...

I suspect there's a conceptual problem here

to what extent is "retain the logical ability" meaningful without attaching it to some knowledge

> to what extent is "retain the logical ability" meaningful without attaching it to some knowledge

Some knowledge is obviously required, I'm just less sure that a specific task like coding benefits all that much from having Shakespeare in the training set...

That’s the thing. LLMs don’t have any logical ability. Only predictive ability. They’re not the same. And that’s why LLMs are a) unreliable and b) not a path to AGI.

How do you know that human logic is any more/better/qualitatively different from the LLMs "predictive ability"? It's already pretty easy to find humans that are strictly worse at reasoning and logic than a decent LLM.

> It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory.

The A in 26B-A4B is the active weights.

The problem is that this is a per-token load/unload at best, not for the whole prompt.

The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.

You're kinda describing the MoE architecture; you can offload expert layers and stream them as-needed if the experts are small enough and the SSD is fast enough.

Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.

The issue is that the MoE knowledge is LLM "knowledge" and it still has a cost so, overall, it has a lower quality/cost ratio.

What he's envisioning is a dense 1B model that looks at the Python specification and your prompt and goes:

> Ah, I get it now! This is like Darmok and Jalad at Tanagra!

Or at least:

> Android UI development in Python? It's UNIX, I know this!

We do work like this sometimes but in general we rely on internalized knowledge so I don't know to what extent it is a viable strategy.

You're right, but the "just" in "just finetune" is doing _a lot_ of work here.

It's still early days and we "just" don't really know how to do it well.

I mean, that's fair, I guess what I mean is, it feels like we're re-using well known solutions even if it takes a bit of effort to re-apply them into how we run inference (and maybe training as well). It will be interesting to see a lot of these approaches compound into anyone with a reasonable GPU or even a Mac running a model much larger than their machine can handle.

We did something similar - Streaming experts. Maintaining an expert cache, optimizing it to simulate running a multi-model agentic workflow on a 2-DGC Spark Cluster. The models we ran were: DeepSeek V4 Flash, Gemma 4 26B A4B, and Nemotron 3 Nano Omni 30B NVFP4. The results were very encouraging in terms of performance and model switching. Check it out here - https://woolyai.com/ai-compute-software/dgx-spark-inference-...

The benchmarks are hidden behind a sign up form. Why not just keep it open?

This looks as if you are just advertising.

It looks? It is advertising :-)

The problem is that knowledge in a LLM is separated in the mathematical sense (vector direction) but not necessarily meaningfully grouped in the matrix (would be easier to split between disk/memory) if such.

I think a rough analog is that it would be difficult to organize the rows of a hash table of everybody in a country by their geographic location.

At this point, people are just thrilled LLMs can even function as they do…

Games used to render entire map in map in memory. Later they figured only the surroundings can we rendered. We can hopefully get same in AI

Indeed, it begs the question why we have "everything" models where instead we could have very efficient "something" models. Typical LLMs out there can generate code and translate between 60 different languages. Sometimes I only need the first part, sometimes the second. Two distinct models would be a lot smaller and run much faster (token-wise).

But they'd be stupider.

The results for English and Python are much better because the model is also trained on Mandarin and Greek and Lisp even if you never make a request or receive a response in Mandarin, Greek or Lisp.

That's news to me since it's unlikely most of those weights are activated when responding to a coding prompt. Can you point to a source/paper that validates this claim?

It's well known and well documented in AI research.

Instead of taking syntactic shortcuts, the richer abstractions learned from multiple languages, and code, and math, and images and audio, ultimately make English comprehension and reasoning far stronger.

If you want citations, ask your favorite LLM how linguistic diversity prevents "surface memorization", overfitting on surface-level English patterns instead of representing the deeper concepts in latent vector space.

One key thing with MoE I am still not understanding is why we don't keep the same expert in memory for a larger number of tokens than 1. Why do we route to some other expert every token? Wouldn't it be more memory efficient to generate at least 2,3,4,8,12 or 32 tokens and then swap the experts?

Expert choice actually happens per layer, not just per token. It's not a limitation when doing inference at scale since all experts are then loaded in fast VRAM anyway. It's mostly just a way to enforce some kind of model sparsity and save on compute.

The new version of Apple Foundation Model (AFM) Core Advanced is an exception, it actually routes experts per prompt (with sporadic rerouting perhaps?) which is more in line with what you're proposing. But this will theoretically make the model less smart than a similar one where experts are picked per layer.

Apple does something similar with their latest foundation model. They process input prompt and based on results they preload required experts. Quite neat solution for the edge devices

Putting the whole model in memory is far faster then swapping to disk.

For local inference the cost of "speed" is not that bad I would think? I wouldn't mind a bit of a delay if it means I can run much larger models on my Mac.

It's pretty painful to have speeds < 30 tok/sec though. Especially if you're used to API providers at higher speeds. It makes any interactive work almost impossible to do efficiently because you have no choice but to context switch after every request.

I assume it will get better over time, and does it improve in speed if you use a larger buffer? Say instead of 2GB you go with 6GB? I imagine it would, and you might need to stream drastically less no?

Yeah, correct! You can set up this engine to get more expert cache slots (e.g 32 instead of 16) to get a better hit rate and better tok/s. it will be 3.5gb instead of 2gb.

You don't know whether King Charles or 42 is the answer to "the Question"

my 2c on King Charles vs 42.

I don't think the problem is that memory footprint can't be reduced. The problem is that the token generation rate is simply far too low.

OP suggests that the token rate of of their solution is ~5 per second. That's at least an order of magnitude slower than commercially available models.

> Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about.

The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.

Anthropic's API has two nines availability and Claude Code is a TUI made with React that can regularly consume more than 1GB of RAM, and the codebase is utter slop. They couldn't fix the flickering bug for over a year!

And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol).

It's not about the people there being smart or not, it's about their and the company's priorities, resources and what they choose to focus on.

One is the UX, which they don't care about because people use their models anyway.

The second would be hardware savings on the order of tens of billions of dollars if they were stupid not do try all possible optimizations.

Spot the difference.

You cannot just "try all possible optimisations". It takes time, effort, and money that could otherwise be spent elsewhere (especially for training, where each training run is especially costly, and optimisations might be promising early on, but cause the final performance of the model to be worse). You need smart people interested in unglamorous work, and if you're swimming in VC money, it's far more straightforward to just throw more GPUs at the problem.

Reply on news.ycombinator.com