Hacker News Reader

Comment by psyklic | Hacker News Reader

Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). For example, their method in the recent paper "Language models can explain neurons in language models" [1] is:

1. Using GPT-4, generate a text explanation of a neuron's activations on sample input.

2. Using GPT-4 again, use the text explanation to simulate the neuron on some new text input.

3. Compare the result to the actual neuron's activations on the new text input.

They justify this by saying human contractors do equally poorly at coming up with text descriptions. However, the procedure is such a black box that it is difficult to make scientific conclusions from the results.

[1] https://openai.com/research/language-models-can-explain-neur...

View on HN

> Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators).

The way OpenAI used GPT-4 is fundamentally different than how GPT-4 was used to score the answers to the MIT exam. In OpenAI's case, they had GPT-4 generate an explanation of when a neuron in GPT-3 would fire. They then gave that explanation back to GPT-4 and had GPT-4 predict when the specific neuron in GPT-3 would fire. The scoring was done by computing the correlation between when GPT-4 predicted the neuron would fire and when it actually fired. The scoring was not done by GPT-4 as was done for the MIT exam

In addition OpenAI did have human evaluators score the explanations as well to make sure they were human interpretable[0]

[0] https://openaipublic.blob.core.windows.net/neuron-explainer/...

Correct except they did this for gpt2 not gpt3

Indeed, the OpenAI paper is more well-founded since the "ground truth" was not generated by GPT-4.

However, both papers rely on black boxes instead of well-understood procedures. This places the papers on weaker scientific footing. For example, a poor explanation/simulation of a neuron's behavior may simply be a consequence of GPT-4. Instead, a scientist would want to "prove" some form of unexplainability.

To do this, a researcher would not use a human at all to explain neuronal behavior. Instead, a simple repeatable algorithm such as topic modeling would be applied. This would lead to significantly stronger scientific conclusions about the neurons. It also proves it is not possible to explain the neuron in some specific sense.

An interesting follow-up to the OpenAI paper might be to quantify how much "more powerful" its textual descriptions are than simpler, well-understood techniques such as topic modeling. That could at least reinforce its use.

I mean that’s literally the purpose of that paper: to show the capabilities of a blackbox (gpt4).

Can't we just have GPT-4 make the scientific conclusions from these results? /s

I don't understand your objection. Step (3) is the one that actually assesses how well the proposed description works, and that is a comparison with the 'real' ground truth.

Reply on news.ycombinator.com