Hacker News Reader

Comment by HawtAds | Hacker News Reader

Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer?

> The distillation data also did not contain any China-sensitive content.

This is a very big disclaimer.

It's like if I generate a dataset focusing exclusively on forestry and arboriculture obviously there won't be any useful censorship, or at least little that can be classified above a statistically significant threshold.

If you want to do a study on something more interesting and useful, do a piece on the various guardrail models of all the major LLM API providers. There are usually both input and output guardrails, and they tend to be almost-black boxes from the model routing point of view.

View on HN

We discuss this in the writeup. While we expected this result, it is important for there to be data backing the claims, and an experimental setup that mirrors productions tasks is a useful tool for the conversations going on about this.

Yeah, the benefit of showing this seems obvious to me. I probably would've expected the censorship to transfer slightly given the anthropic owl paper from years ago https://alignment.anthropic.com/2025/subliminal-learning/

But that was about transferring from a finetuned model to another finetune of the same base model, good to see more evidence that it doesn't transfer cleanly across different base models in a more realistic scenario than an "owl-loving model"

Edit: From *year ago. It's been a long year haha

Sometimes it's important to rigorously investigate and prove "obvious" things. Sometimes those things turn out to not be obvious. That's part of good science.

It is surprisingly not obvious, neural nets are weird just like brains

https://www.nature.com/articles/s41586-026-10319-8

Subliminal Learning: language models transmit behavioral traits via hidden signals in data

https://arxiv.org/html/2507.14805v1

"In our main experiments, a “teacher” model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a “student” model trained on this dataset learns T. This occurs even when the data is filtered to remove references to T."

In those experiments, the effect only happened between models from the same family (and likely the same weight initialization).

> Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer?

If the model architecture is too similar it can transfer, even if you never sample on it... because the censorship can depend on hidden state in common that you do sample.

This creates a particular value in distinct architectures that isn't apparent from their direct utility.

It is self-evident. But now if someone questions it in the future, this is a data point that distilling from Deepseek isn’t going to turn your finance bot into a communist.

Also, sometimes things which seem self-evident turn out to be surprising.

Reply on news.ycombinator.com