Hacker News Reader

Comment by az226 | Hacker News Reader

So lame they focus on gatekeeping the exams and crying about permission as opposed to challenging the paper’s shit methodology.

View on HN

I mean if the dude submitted the work without the permission of the actual lead authors that's a pretty big violation, which also could explain in part why the methodology was so bad.

But that’s something they should deal with internally. It is quite strange actually. The statement should also come from MIT’s public affairs office and not the professors themselves. So it looks like they are big mad and acting on their own to save face.

The person who submitted to arxiv (and promoted the paper on Twitter) is one of the senior authors (listed last in the author block), so I don’t think that’s an excuse for publishing to arxiv before doing due diligence on methodology

Apparently some of the other profs were not in the loop about the arxiv submission though

Senior author is a euphemism for "happens to run the lab". Explains why the other "senior authors" are lashing out, they never expected having to know what a paper that bears their name is all about.

Sure - the point is that the preprint wasn’t posted by a student without permission of their advisor, it was posted by a professor without (apparently) permission of their collaborators, who aren’t part of a monolithic lab for what it’s worth.

However, I generally agree with your take, and the response seems to me to kind of dodge the responsibility for methodology issues. that responsibility is not really compatible with the “hey we’re submitting to arxiv in a few days, last chance for comments” approach to collaboration that seems to be the minimum expected bar for signing off on the submission form that all authors are aware and agree to publication

Posted by a lecturer, not a professor I believe.

He’s an associate professor at BU, and I guess a lecturer at MIT

That's nonsense. Senior authors may not contribute text to the paper but they will normally be involved in every other way - reading drafts, making recommendations, directing the research.

I know it's standard on HN to accuse senior academics of exploiting their PhD students and I'm pretty sure there is plenty of that to go around but it is a very rare PhD student that can write a publishable paper without advise and guidance from their advisor. The name gives a hint, even.

I'm speaking in this as a recently graduated PhD student btw. I don't have any conflicts of interest (well, not yet, hopefully). It sucks that many students have absolutely rotten relations with their advisors but that's exactly because the student depends so much on the advisor for direction that it's easy for the power differential to be exploited by unscrupulous individuals.

The main thrust of their complaint (as paper authors!) is that they used the "copyrighted" exam questions. I don't understand how both can be true: to have contributed to this paper, which entirely rests on ChatGPT (not) answering exam questions, and not knowing that that is what the paper is all about.

That is certainly an issue that would be discovered well before anyone is sitting down to write the actual paper.

This may be true in some fields but it's generally not the case in academic computer science. (Generally.)

>Apparently some of the other profs were not in the loop about the arxiv submission though

Sounds more like they're throwing the one senior prof under the bus to protect their own faces; saying 'I didn't agree to preprint submission' would be the first excuse I'd come up with.

At that point, they had been majorly involved in the research, if they were not involved in the preprinting they must have seen several drafts at least, at which point they could've asked to get taken out. But they didn't do that, and that's telling.

I tend to agree, but it’s hard to know exactly how it went down from our outsiders perspective. They’re implying that they asked for changes before submission, and that they weren’t given a final call to ask for their names to be removed if it came to that. Personally I would be less skeptical about this version of events with more transparency about whatever technical issues they had with the manuscript

The paper is in Neurips template and was possibly submitted to the conference. And now other authors are complaining when other people pointed to it.

You... think it's lame that an academic institution would focus on protecting their students and professors?

Paper didn’t include the exams themselves, please correct me if my understanding is incorrect.

Figure 4 on the last page of the PDF might include a single question.

Based on what i watched by Yannick on YouTube and haven't actually read... Questions weren't part of the paper but authors made a goof and included all of them in GitHub. Shortly after they deleted these questions but as part of a new commit so one could see the change, thus all the questions.

But then this dumb statement should have been taking about the upload to GitHub and not the paper and how collaborators weren’t aware, didn’t get permission to run the analysis, etc. Sure, they’re all connected to the paper but they’re separate issues.

The signatories are some of the paper's authors. It's not common for authors to criticise their own methodology as shitty. Anyway it's clear they didn't think the paper was ready to be made public so what is there to criticise about its methodology? It's a work in progress.

You realize this is only one part of the response?

> We want to emphasize that all the student authors in this paper worked really hard on what could have been a very interesting and valuable paper had the data been collected with consent. The many problems with the published work were not the fault of the students.

This is the conclusion of the memo. The problems with methodology are clearly a secondary concern. They also seem to imply that issues with consent are solely or mainly the cause of the methodology problems. This could be true, but idk.

This is my opinion as an MIT grad who still views OCW content and similar content from other universities from time-to-time.

(Complete speculation: I almost feel as if certain professors or lecturers are “embarrassed” about their content.)

Working hard does not necessarily mean getting it right.

(Also an mit grad).

No one said it did. Did you mean to reply to someone else?

This thread was about whether the response focused on the consent or methodology problems and it’s my opinion that they are focusing on the consent issues more.

I only copied a quote from the memo.

> He did so without the consent of many of his co-authors and despite having been told of problems that should be corrected before publication.

I think they are point out consent issues in multiple cases, but also talk about "problems that should be corrected before publication".

I can't really see how there would problems to be corrected before publication.

If it's only a consent issue but the rest of the methodology is sound, they should just say "Drori should have waited for the consent to go ahead with publication, but the paper is fine"?

To clarify, when I said this was the _conclusion_ of the memo, I meant that it was the literal closing statement (besides the one-liner/quip about GPT-4 not being able to earn an MIT degree).

I didn’t mean that it was the be-all and end-all of those CSAIL profs’ perspective(s).

The wording is still very suspect to me and seems like the Writing Center would have some constructive feedback for them (unless the meaning of words doesn’t matter).

I take the stance that getting it right is implicitly understood to be primary, as it should be in science

I agree but not everyone thinks like this even at MIT, as you should know since you said that you’re an alum.

Precisely.

Reply on news.ycombinator.com