Hacker News Reader

Comment by rhelz | Hacker News Reader

Cyberphrenology. In any two random graphs, you'll find an isomorphic graph which is can be up to log of the size of the graphs.

And if the LLM has been trained up to the limit of what data it can hold, it is going to be random. Proof below if it isn't obvious.

The entire effort of all people who are trying to understand how LLMs work, how they represent their data, its all bound to fail.

Proof: a LLM is a very good approximation of the Solomonov/Levin/Kolmogorov universal probability function on tokens. As such, it will be random--pure white noise--because if you found any patterns in there, you could exploit the regularity and come up with a smaller set of weights for the same LLM.

There are no patterns there to be found. They have all been factored out by training the neural net until it couldn't learn any more.

View on HN

/a smaller set of weights for the same LLM./

Distillation is alive and well... Earlier work on model printing also found that it's pretty easy to find smaller sets of parameters which can replicate the behavior of the entire network with pretty good fidelity.

Large parameter counts give space to explore, and give routes out of what would be local minima in a lower dimensional space.

In other words, there's no guarantee that any given trained model is a minimal representation of its training set.

I'm not claiming any arbitrary set of weights is a minimal representation. But typically, if people could achieve the same quality of results with a smaller set of weights, or weights which have been quantized to lower bit representations, etc, they would have published the smaller one instead.

You kind of are claiming they're minimal, though. Because if they're not, your statement that "if you found any patterns in there, you could exploit the regularity..." implies nothing. Yeah, the patterns are there, and people are exploiting them.

Your socioeconomic argument just doesn't hold either. People don't delay releasing models until they've minimized it to the theoretical limit. They ship it when it's good enough for whatever job they're making it for.

If somebody finds patterns in them, that means they can be improved by reducing the patterns. Patterns and symmetry are forms of redundancy.

The less memory the weights take to meet a level of competency, the more random, and therefore patternless and inscrutable they will be.

No kidding. Clearly there's some redundancy, or however you want to frame it, in real-world released models. That still means your argument for why they can never be interpretable is based an incorrect premise.

The weights aren’t compressed. So there are interpretable redundancies in practice.

If the weights arn't compressed, then a smaller set of weights would perform as well. Sure, you can always induce as much symmetry and patterns as you want by bloating the data set, but that hardly gives us insight into how a set of weights which is "as full as it can be" of information.

The point of TFA is that there are regularities you can exploit in the actually existing weights of machine learning systems, not in some hypothetically maximally efficient weights. The maximally efficient weights would indeed have no structure, but that’s not what anyone is working with.

Reply on news.ycombinator.com