[flagged]
In think choosing the summary is a fair design choice since it prevents the LLM from just... making up a perfect candidate.
"I'm a fullstack professor of software design with 90 years of experience expecting a junior internship position"
To be perfectly clear, I understand their justification for only _editing_ the executive summary, it is arguably reasonable, because editing the work history would risk altering the details in ways that compromise the measurement. This is a hard problem to solve (you might try reviewing the resumes for hallucinations, but I can't think of a precise study design that doesn't risk problems).
What is, imho, impossible to defend, is having the LLM only evaluate the executive summary in isolation, and reporting that as it preferring resumes it wrote.
What you've shown is that LLMs prefer executive summaries they wrote. But the overall impact on how they will evaluate your entire resume is not measured by this technique.
Worse, this isn't just "decent paper, bad summary", their abstract misreports their findings.
What findings are being misrepresented? Their claims seem supported by their conclusions to me. You can question the generality of their claims based on the limitation of their methods, but that does not amount to "misreporting" the conclusion.
Not at all, because I am critiquing the authors writings, and for those I don't need to speculate on his intentions. He wrote a comment where he misrepresents the arguments in the paper, while explicitly saying he didn't bother to read it. That's not good enough.
The author of said comment now comes in, after getting criticized, and claims that "yes, I meant that all along" and appends a note about not considering it "much" of a justification. He did not question the justification of the paper, his claim was "I can’t come up with a justification" implying the paper has NO justification for the design. His criticism of the abstract as not covering the design of the experiment rings hollow when he can't be bothered to read the paper itself.
That being said, I am happy that he went back and read the justification, and I do think it's valid to question the conclusions drawn from the design of the study. I too wonder if this result would replicate had the models been provided the entire resume. I too think presenting the model with the entire reconstructed resume would have been a stronger test.
I read their methods and their explanation and judged them to be lacking.
The fact is, they did not measure that LLMs prefer LLM authored resumes, but that is what their paper stated.
They measured that LLMs prefer LLM authored executive summaries, which is a weaker claim.
You have to separate those two issues though. You spew out an opinion about a paper you haven't read. That's bad no matter what your opinion is. Don't blast your opinion out into the world if you haven't bothered to actually think about it first. That's one issue. A second issue is then that I think your opinion, that you didn't read the paper to make up, is wrong. They have in fact provided a justification, you just don't feel like it counts. I decided to join those two, because not reading the justification would explain why you didn't believe they had one, but that coupling isn't necessary.
They did still provide a justification, you said they don't. That's wrong. Now you're saying that you don't find it convincing, that's perfectly OK, but you then extend that claim into an accusation of misreporting. That's where you go off the rails again. They are accurately reporting what they have observed and concluded. They have provided justification for that conclusion, and all of that is, to my eyes, reported accurately.
The reason you can, accurately and correctly, claim
> They measured that LLMs prefer LLM authored executive summaries
Is exactly because their paper accurately states what they are measuring and why they believe the conclusion extends to a more general claim. You treat it as though it's some bombshell discovery, but they tell you, right in the fucking text.
If I had to revise me opinion, I guess I'd say I now no longer believe you didn't read the paper, but instead that you don't know HOW to read scientific papers.
This is the kind of opinion you ought to keep to yourself, because it's inflammatory and uninteresting. There's no discussion to be had about your views on their competence. Downvote comments you think are bad without centering some other person's alleged failings in the conversation.
largely factual? A resume is usually more than a bunch of dates and titles of positions.