Hacker News Reader

Comment by DennisP | Hacker News Reader

This looks like a big advance in alignment research. A big problem has been that LLMs were just a giant set of inscrutable numbers, and we had no idea what was going on inside.

But if this technique scales up, then Anthropic has fixed that. They can figure out what different groups of neurons are actually doing, and use that to control the LLM's behavior. That could help with preventing accidentally misaligned AIs.

View on HN

To me, it sounds more like a good lead for pruning.

> We find that the features that are learned are largely universal between different models, so the lessons learned by studying the features in one model may generalize to others.

Hm. I wish they'd said more about that. Does that mean they found the same feature recognizers when training with the same training set? Or what? This tells us something, but what does it tell us?

Some architectures are relatively well understood. Eg in CNNs, the first layers detect low level features like edges, gradients, etc. The next layer then combines these features to more complex structures like corners or circles. Next layer will combine these features to even higher level features and so on. [1]

Typically, you can take a pre-trained model and retrain it on your new dataset by only changing the weights of the last layer(s).

Some loss functions even measures the difference between the high-level features of two images, typically extracted from a pre-trained CNN (Perceptual Loss).

[1]Matt Zeiler did an amazing work on these findings 10 years ago (https://arxiv.org/abs/1311.2901).

Reply on news.ycombinator.com