Hacker News Reader

Comment by yorwba | Hacker News Reader

That is the actual paper, it's published on transformer-circuits.pub.

It's not peer-reviewed?

Google Scholar claims 380 citations, which is, I think, a respectable number of peers to have reviewed it.

Unless it’s part of a link review farm. I haven’t looked, and you are probably correct; but I would do a bit of research before making any assumptions

That's not at all how peer review works.

It's not how pre-publication peer review works. There, the problem is that many papers aren't worth reading, but to determine whether it's worth reading or not, someone has to read it and find out. So the work of reading papers of unknown quality is farmed out over a large number of people each reading a small number of randomly-assigned papers.

If somebody's paper does not get assigned as mandatory reading for random reviewers, but people read it anyway and cite it in their own work, they're doing a form of post-publication peer review. What additional information do you think pre-publication peer review would give you?

peer review would encourage less hand wavy language and more precise claims. They would penalize the authors for bringing up bizarre analogies to physics concepts for seemingly no reason. They would criticize the fact that they spend the whole post talking about features without a concrete definition of a feature.

The sloppiness of the circuits thread blog posts has been very damaging to the health of the field, in my opinion. People first learn about mech interp from these blog posts, and then they adopt a similarly sloppy style in discussion.

Frankly, the whole field currently is just a big circle jerk, and it's hard not to think these blog posts are responsible for that.

I mean do you actually think this kind of slop would be publishable in NeurIPS if they submitted the blog post as it is?

"peer review would encourage less hand wavy language and more precise claims"

In theory, yes. Lets not pretend actual peer review would do this.

So you think that this blog post would make it into any of the mainstream conferences? I doubt it.

IME: most of the reviewers in the big ML conferences are second-year phd students sent into the breach against the overwhelming tide of 10k submissions... Their review comments are often somewhere between useless and actively promoting scientific dishonesty.

Sometimes we get good reviewers, who ask questions and make comments which improve the quality of a paper, but I don't really expect it in the conference track. It's much more common to get good reviewers in smaller journals, in domains where the reviewers are experts and care about the subject matter. OTOH, the turnaround for publication in these journals can take a long time.

Meanwhile, some of the best and most important observations in machine learning never went through the conference circuit, simply because the scientific paper often isn't the best venue for broad observation... The OG paper on linear probes comes to mind. https://arxiv.org/pdf/1610.01644

Of the papers submitted to a conference, it might be that reviewers don't offer suggestions that would significantly improve the quality of the work. Indeed the quality of reviews has gone down significantly in recent years. But if Anthropic were going to submit this work to peer review, they would be forced to tighten it up significantly.

The linear probe paper is still written in a format where it could reasonably be submitted, and indeed it was submitted to an ICLR workshop.

What? Yes it is! This is exactly how peer review works! People look at the paper, read it, and then reproduce it, poke holes, etc.

Peer review has nothing to do with "being published in some fancy-looking formatted PDF in some journal after passing an arbitrary committee" or whatever, it's literally review by your peers.

Now, do I have problems with this specific paper and how it's written in a semi-magical way that surely requires the reader suspend disbelief? For sure, but that's completely independent of the "peer-review" aspect of it.

If you believe that citation is the same as review, I have stuff to sell you.

Reviewing a paper can easily take 3 weeks full time work.

Looking at a paper and assuming it is correct, followed by citing it, can literally take seconds.

I'm a researcher and there are definitely two modes of reading papers: review mode and usage mode.

No, I don't believe that a citation is the same as peer review, it is, as I understand it from the post you're replying to, an observation that other people have looked at it. (Indeed, if you actually go through and _read_ this particular paper, you can see that they also cite multiple reproductions and extensions of the paper. I would call that peer review—and certainly much better peer review than most nonsense that appears from committees. Ask me how I know.)

How would you peer-review something like that? Or rather, how would you even _reproduce_ any of that? Some inception-based model with some synthetic data? Where is the data? I'm sure the paper was written in good faith, but all it leads to is just more crackpottery. If I had a penny for each time I've heard that LLMs are quantum in nature "because softmax is essentially a wave function collapse" and "because superposition," then I would have more than one penny!

There are several [replications and comments from reviewers](https://transformer-circuits.pub/2022/toy_model/index.html#c...), so yes, it is.

Would that help that much nowadays when peer-reviewing is just 2 students in a trench coat? In my opinion, especially on more novel works like this, the discussions between people in the field are way more useful than a stamp of approval from whatever journal.

Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications.

A few thoughts:

(1) As others have commented, I think peer review in ML is pretty widely accepted to be dysfunctional right now. I think most people who have published in ML conferences would agree. It's not unusual for early PhD students and sometimes even undergrads to review, and reviewers are overburdened to the point where they can carefully consider all their papers. Everything I've said so far is just anecdote and opinion though. A more objective test was the NeurIPS 2021 Consistency Experiment ( https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste... ) which found that if a paper was accepted by the conference, there was only a ~50% chance that a parallel review process would come to the same conclusion.

(2) Modern peer review is a relatively modern invention, arising in post-WW2 science as the scientific community grew dramatically, and there was a need for more systematized ways to make decisions about publication, funding, jobs, etc. Famously, Einstein was offended by one of his papers being sent for review. I don't think it's at all obvious that this transition has been good for science! I see lots of people writing papers for reviewers, rather than with the goal of doing the most impactful science they can.

(3) As background, I spent 5 years of my life running a scientific journal ( https://distill.pub/ ), trying to have excellent review processes and enable non-traditional papers to be peer reviewed. I honestly just burnt out on this. Now I just want to do good research.

(4) We do circulate draft papers to researchers working on similar topics at other industry groups, and in academia. As other comments have noted, this sometimes leads to public comments on our papers. In many cases, these are a much deeper review than you'd see in typical peer review processes, such as independent reproduction of experiments.

Reply on news.ycombinator.com