One of my papers got declined today
mathstodon.xyz
mathstodon.xyz
(That paper has now been cited 971 times according to Google Scholar, despite never appearing in a journal.)
As its author noted, the paper has done fine ciation- and impact-wise.
Your link is the website I put up for non-experts when I announced the issue.
If the course website was even on the open web to begin with. If they're in some university content management system (CMS), chances are that access is limited to students and teachers of that university and the CMS gets "cleaned" regularly by removing old and "unused" content. Let alone what will happen when the CMS is replaced by another after a couple of years.
https://wiki.archiveteam.org/index.php/University_Web_Hostin...
regards...
It's a good initiative. Next step: everybody realizes that researchers are just random people like everybody. Maybe that could kill any remaining imposter syndrome.
A rejection, although common, is quite tough during your PhD though, even ignoring the imposter syndrome, because in a short time, you are expected to have a bunch of accepted papers, in prestigious publications if possible. It feels like a rejection slows you down, and the clock is still ticking. If we could kill some of this nefarious system, that'd be good as well.
If you read the full post he’s making the exact same point as you: it’s common and normal to get a paper rejected even if you’re Terence Tao, so don’t treat a rejection like the end of the world.
Oh?
Perelman comes to mind (as the only person who has been eligible to claim one of the Millennium prizes), although he is no longer actively practicing math AFAIK. Of Abel prize winners, Wiles proved Fermat's last theorem, and Szemeredi has a number of number-theoretic and combinatorial contributions.
Recently deceased (past ~10 years) include figures such as John Nash, Grothendieck, and Conway.
Tao is definitely one of the most well-known mathematicians, and he's still got several more decades of accomplishments ahead of him, but I don't know that he rises to "greatest living mathematician" at this point.
That said, I do appreciate that he indicates that even his papers get rejected from time to time.
I didn't know :-)
> If you read the full post he’s making the exact same point as you
Oh yeah, that's because I did read the full post and was summarizing. I should have made this clear.
I think it's important to post a follow-up comment clarifying that papers are reviewed following a double blind peer review process. So who the author is shouldn't be a factor.
Also, the author clarified that the paper was rejected on the grounds that the reviewer felt the topic wasn't a good fit for the journal. This has nothing to do with the quality of the paper, but uploading editorial guidelines on the subject. Trying to file a document in a wrong section and being gently nudged to file under another section hardly matches the definition of a rejection that leads authors to question their life choices.
having higher reputation means higher responsibility not to crush someone with it in the sub-fields you aren't as proficient as
I'm sure he's a genuinely nice, friendly person trying to do the right thing. But he is also likely confident as hell and never felt like an imposter anywhere.
The point he is making is all the motr convincing especially that he is seen as very good, whether he had imposter syndrome or not.
I once wrote a paper along the lines of "look we can do X blazingly fast, which (among other things) lets us put it inside a loop and do it millions of times to do Y." A reviewer responded with "I don't understand what the point of doing X fast is if you're just going to put it in a loop and make it slow again." He also asked us to run simulations to compare our method to another paper which was doing an unrelated thing Z. The editor agreed that we could ignore his comments.
Not so much the fifth!
I hear this all the time, but this is actually a real phenomenon that happens when well-known senior figures are rightfully cautious about over-citing their own work and/or are just so familiar with their own work that they don't include much of it in their literature review. For everybody else in the field, it's obvious that the work of famous person X should make up a substantial chunk of the lit review and be explicit about how the new work builds on X's prior literally paradigm shifting work. You can do a bad job at writing about your own past work for a given audience for so many different reasons, and many senior academics do all the time, making their work literally indistinguishable from that of graduate students --- hence the rejection.
I may be misremembering, but I believe the case with Grant was that the referee was using his own work to discredit his submission. Ie "If the author was aware of the work of Adam Grant, they would understand why the submitted work is wrong."
If, as a developer, you had the experience of looking at some terrible code, angrily searching for whoever wrote that monstrosity, only to realize that you did, that's the idea.
By the way, even though some of you may know about it, here's the link to the Journal of Negative Results: https://www.jnr-eeb.org/index.php/jnr
I had the lucky opportunity to do a postdoc with one of the most famous people in my field, and I was shocked how much difference the name did make- I never had a paper rejection from top tier journals submitting with him as the corresponding author. I am fairly certain the editors would have rejected my work for not being fundamentally on an interesting enough topic to them, if not for the name. The fact that a big name is interested in something, alone can make it a "high impact subject."
At least it indicates that the system is working somewhat properly some of the time...
This feels like a superhuman trying to empathize with a regular person.
That said, given the labor involved in academic publishing and review, the optimal rejection rate should be quite low, i.e. find a lower cost way to pre-filter papers. OTOH, the reviewers may get value from rejected papers...
[1] least publishable unit
These days I read a lot of CS papers with an eye on solving the problems and personally I tend to find the short ones useless. (e.g. pay $30 for a 4-page paper because it supposedly has a good ranking function for named entity recognition except... it isn't a good ranking function)
But I think the more important point is that very few people are capable of publishing papers in top math journals.
I've found similar insights when I joined a community of musicians and also discovered twitch / youtube presences of musicians I listen to. Some of Dragonforces corona streams are absolutely worth a watch.
It's easy to listen to mixed and finished albums and... despair to a degree. How could anyone learn to become that good? It must be impossible, giving up seems the only rational choice.
But in reality, people struggle and fumble along at their level. Sure enough, the level of someone playing guitar professionally for 20 years is a tad higher than mine, but that really, really perfect album take? That's the one take out of a couple dozen.
This really helped me "ground" or "calibrate" my sense of how good or how bad I am and gave me a better appreciation of how much of a marathon an instrument can be.
What publication in a journal gives you is context, social proof, and structured placement in public archives like libraries. This remains true in the age of the Internet.
That said, I do think that "publish or perish" plays an unspoken role here. I see a lot of colleagues trying to push out "least publishable units" that might barely pass review (by definition). If you need to juice your metrics, it's a common strategy that people employ. Still, I think a lot of papers would pass peer review more easily if researchers just combined multiple results into a single longer paper. I find those papers to be easier to read since they require less boilerplate, and I imagine they would be easier to pass peer review by the virtue that they simply contain more significant results.
Yes, a longer paper puts more work on the peer reviewers (handful of people). But splitting one project in multiple papers puts more work on the reader (thousands of people). There is a balance to strike.
To me, the point of peer review is to both evaluate the science/correctness of the work, but also to ensure that this is something novel that is worth telling others about. Does the manuscript introduce something novel into the literature? That is my standard (and the standard that I was taught). I typically look for at least one of three things: new theory, new data/experiments, or an extensive review and summation of existing work. The more results the manuscript has, the more likely it is to meet this novelty requirement.
Peer review misses 100% of long papers that don't get submitted, where multiple non-outrageous claims were too much work to develop.
Authors start at an attainable stretch goal, hope for a quick rejection if that’s the outcome, and work their way down the list. That’s why rejection is inevitable.
You may have to leave a year of work on arxiv, with the expectation that the work will be rehashed and used in other published papers.
AirBnB being rejected for funding, musicians like Schubert struggling their entire life, writers like Rowling in poverty.
Rejection will always be the norm in competitive winner take all dynamics.
- it was fine, I desk rejected terence tao, his result was a bit meh and the write up wasn't up to my standard. Then I had a bit of a quite office hour, anyway, ...
I have to not say a word to them as I talk to them or else I could ruin the whole peer review thing!
"Hey honey, I reviewed X work from Y famous person today"
In what sense would it ruin peer review to reveal your role after you already wrote and submitted the review?
It could support links/backref, citations(forks), questions(discussions), tags, followers, etc easily.
Having a curated list can be important to separate the wheat from the chaff, especially in an era with ever increasing rates of research papers.
Curated lists can also exist on the site. Look at awesome* repos on github eg https://github.com/vinta/awesome-python
Obviously, some lists can be better than the others. Usual social mechanics is adequate here.
I don't think peer-review has to be done by journals, I'm just not sure what the better solution is.
Nothing prevents the site introducing more direct peer review (published X papers on a topic -> review a paper).
Though If we compare two cases: reading a paper to leave an anonymous review vs reading a paper to cite it. The latter seems like more authentic and useful (less perversed incentives).
Imagine, you found a paper on arxiv-like site: there can be metadata that might help determine quality (author credentials, citations by other high-ranked papers, comments) but nothing is certain. There may be cliques that violently disagree with each other (paper clusters with incompatible theories). The medium can help with highlighting quality results (eg by choosing the default ranking algorithm for the search, introducing StackOverflow-like gamification) but it can’t and shouldn’t do science instead of practitioners.
"""
... I once almost solved a conjecture, establishing the result with an "epsilon loss" in a key parameter. We submitted to a highly reputable journal, but it was rejected on the grounds that it did not resolve the full conjecture. So we submitted elsewhere, and the paper was accepted.
The following year, we managed to finally prove the full conjecture without the epsilon loss, and decided to try submitting to the highly reputable journal again. This time, the paper was rejected for only being an epsilon improvement over the previous literature!
...
"""
Instead of allowing the community to join forces by breaking up a larger problem into pieces, it encourages siloing and camper mentality.
In the journal's defense though, what most likely happened is that the reviewers were different between submissions and they didn't know about the context. Ultimately, I think, this type of rejection comes down to the mostly the reviewers discretion and it can lead to this type of situation.
I cut off the rest of the post but Tao finished it with this:
"""
... Being an editor myself, and having had to decline some decent submissions for a variety of reasons, I find it best not to take these sorts of rejections personally,
...
"""
The anecdote about the highly reputable journal rejecting the second of a 2-part paper which (presumably) would have been accepted as a 1-part paper is telling.
> With hindsight, some of my past rejections have become amusing. With a coauthor, I once almost solved a conjecture, establishing the result with an "epsilon loss" in a key parameter. We submitted to a highly reputable journal, but it was rejected on the grounds that it did not resolve the full conjecture. So we submitted elsewhere, and the paper was accepted.
> The following year, we managed to finally prove the full conjecture without the epsilon loss, and decided to try submitting to the highly reputable journal again. This time, the paper was rejected for only being an epsilon improvement over the previous literature!
Suppose the full result is worth 7 impact points, which is broken up into 5 points for the partial result and 2 points for the fix. The journal has a threshold of 6 points for publication.
Had the authors held the paper until they had the full result, the journal would have published it, but neither part was significant enough.
Scholarship is better off for them not having done so, because someone else might have gotten the fix, but the journal seems to have acted reasonably.
In some sense, but it does feel like the journal is missing the bigger picture somewhat. Say the two papers are A and B, and we have A + B = C. The journal is saying they'll publish C, but not A and B!
I suspect readers don’t find it as exciting to read partial result papers. Unless there is an open invitation to compete on its completion, which would have a purpose and be fun. If papers are not page turners, then the journal is going to have a hard time keeping subscribers.
On the other hand, publishing a proof of a Millennium Problem as several installments, is probably a fantastic idea. Time to absorb each contributing result. And the suspense!
Then republish the collected papers as a signed special leather limited series edition. Easton, get on this!
Yeah I agree, a partial result is never going to be as exciting as a full solution to a major problem. Thinking on it a little more, it seems more of a shame the journal wasn't willing to publish the first part as that sounds like it was the bulk of the work towards the end result.
I quite like that he went to publish a less-than-perfect result, rather than sitting on it in the hopes of making the final improvement. That seems in the spirit of collaboration and advancing science, whereas the journal rejecting the paper because it's 98% of the problem rather than the full thing seems a shame.
Having said that I guess as a journal editor you have to make these calls all the time, and Im sure every author pitches their work in the best light ("There's a breakthrough just around the corner...") and Im sure there are plenty of ideas that turn out to be dead ends.
If a series of incremental results were as prestigious as holding off to bundle them people would have reason to collaborate and complete each other's work more eagerly. Delaying an almost complete result for a year so that a journal will think it has enough impact point seems straightforwardly net bad, it slows down both progress & collaboration.
Supposing it is, you have to trade off publishing these incremental results against publishing someone else’s complete result.
What if it had taken ten papers to get there instead of two? For a sufficiently important problem, sure, but the interesting question is at a problem that’s interesting enough to publish complete but barely.
That's because every researcher has a hierarchy of journals that they monitor. Prestigious journals are read by many researchers. So you're essentially competing for access to the limited attention of many researchers.
Conversely, publishing in a premium journal has more value than a regular journal. And the big scientific publishers are therefore in competition to make sure that they own the premium journals. Which they have multiple tricks to ensure.
Interestingly, their tricks only really work in science. That's because in the humanities, it is harder to establish objective opinions about quality. By contrast everyone can agree in science that Nature generally has the best papers. So attempting to raise the price on a prestigious science journal, works. Attempting to raise the price on a prestigious humanities journal, results in its circulation going down. Which makes it less prestigious.
This is exactly what people think, and exactly what happens, especially in winner-takes-all situations. You end up with an interesting tension between how long you can wait to build your story, and how long until someone else publishes the same findings and takes all the credit.
A classic example in physics involves the discovery of the J/ψ particle [0]. Samuel Ting's group at MIT discovered it first (chronologically) but Ting decided he needed time to flesh out the findings, and so sat on the discovery and kept it quiet. Meanwhile, Burton Richter's group at Stanford also happened upon the discovery, but they were less inclined to be quiet. Ting found out, and (in a spirit of collaboration) both groups submitted their papers for publication at the same time, and were published in the same issue of Physical Review Letters.
They both won the Nobel 2 years later.
Every night Gauss went to sleep, mathematics was held back a week.
Finally one professor saw what was happening, insisted that Gauss take some time off - being German that involved walking in the woods.
https://en.m.wikipedia.org/wiki/Wiles%27s_proof_of_Fermat%27...
Everybody is free to keep a blog for this kind of informal chat/brainstorming kind of communication. Paper publications should be well-written, structured, thought-through results that make it worthwhile for the reader to spend their time. Anything else belongs to a blog post.
Instead, we should look at which side the, uh, industry currently tends to err. And this is definitely not the "sitting on your incremental results" side. The current motto of academia is to publish more. It doesn't matter if your papers are crap, it doesn't matter if you already have significant results and are working on something big, you have to publish to keep your position. How many crappy papers you release is a KPI of academia.
I mean, I can imagine a world were it would have been a good idea. I think it's a better world, where science journals don't exist. Instead, anybody can put any crap on ~arxiv.org~ Sci-Hub and anybody can leave comments, upvote/downvote stuff, papers have actual links and all other modern social network mechanics up to the point you can have a feed of most interesting new papers tailored specially for you. This is open-source, non-profit, 1/1000 of what universities used to pay for journal subscriptions is used to maintain the servers. Most importantly, because of some nice search screens or whatever the paper's metadata becomes more important than the paper itself, and in the end we are able to assign 10-word simple summary on what the current community consensus on the paper is: if it proves anything, "almost proves" anything, has been 10 times disproved, 20 research teams failed to reproduce to results or 100 people (see names in the popup) tried to read and failed to understand this gibberish. Nothing gets retracted, ever.
Then it would be great. But as things are and all these "highly reputable journals" keep being a plague of society, it is actually kinda nice that somebody encourages you to finish your stuff before publishing.
Now, should have been this paper of Tao been rejected? I don't know, I think not. Especially the second one. But it's somewhat refreshing.
In the software world, it's often desired to have a steady stream of small, individually reviewable commits, that each deliver a incremental set of value.
Dropping a 20000 files changed bomb "Complete rewrite of linux kernel audio subsystem" is not seen as prestigious. Repeated, gradual contributions and involvement in the community is.
I think the impact to scholarship in general is less clear. Do you immediately publish once you get a "big enough" result, so that others can build off of it? Or does this needlessly clutter the field with publications? There's probably some optimal balance, but I don't think the right balance is immediately clear.
Discovering something is hard, proving it correct is hard, and writing a paper about is hard. Why delay all this?
The journal may have acted systematically, but the system is arbitrary and capricious. Thus, the journal did not act reasonably.
You see it in software as well: As a manager in calibration meetings, I have repeatedly seen how it is harder to convince a committee to promote/give a high rating to someone with a large pile of crucial but individually small projects delivered than someone with a single large project.
This is discouraging to people whose efforts seem to be unrewarded and creates bad incentives for people to hoard work and avoid sharing until one large impact, and it's disastrous when (as in most software teams) those people don't have significant autonomy over which projects they're assigned.
Each individual step along the way merely has some rationale, but rationales come in the full spectrum of quality.
I am not sure what's the exact conjecture that the author solved, but if the epsilon difference is between an approximate solution versus an exact solution, and the journal rejected the exact solution because it was "only an epsilon improvement", I might question how reputable that journal really was.
It’s frustrating but the result of a somewhat haphazard process. It’s also not uncommon for conflicting comments within the same review cycle. Some of this may be attributed to a lack of clear communication by the author. But on occasion, it leads me to believe many journals don’t take a lot of time selecting appropriate reviewers and settle for the first few that agree to review.
Are there any mechanisms to balance out the "race to the bottom" observed in other types of academic compensation? e.g. increase of adjunct/gig work replacing full-time professorship.
Do universities require staff to perform a certain number of reviews in academic journals?
I was approached to review something for no compensation as well, but I was a bad fit.
Professors are expected to review by their employer, typically, and it's a (very small) part of the tenure process.
No. Reviewers mostly do it because its expected of them, and they want to publish their own papers so they can get grants
In the end, the university only cares about the grant (money), because they get a cut - somewhere between 30-70% depending on the instituition/field - for "overhead"
Its like the mafia - everyone has a boss they kick up to.
My old boss (PI on an RO1) explained it like this
Ideas -> Grant -> Money -> Equipment/Personnel -> Experiments -> Data -> Paper -> Submit/Review/Publish (hopefully) -> Ideas -> Grant
If you don't review, go to conferences/etc. its much less likely your own papers will get published, and you won't get approved for grants.
Sadly there is still a bit of "junior high popularity contest" , scratch my back I'll scratch yours that is still present in even "highly respected" science journals.
I hear this from basically every scientist I've known. Even successful ones - not just the marginal ones.
Depends on what you mean by "require". At most research universities it is a plus when reviewing tenureship files, bonuses, etc. It is a sign that someone cares about your work, and the quality of the journal seeking your review matters. If it were otherwise faculty wouldn't list the journals they have reviewed for on their CVs. If no one would ever find out about a reviewers' efforts e.g. the process were double blind to everyone involved, the setup wouldnt work.
Indeed, but when someone of Tao's caliber submits a paper, any editor would (should) make an extra effort to get the very best researchers to referee the paper.
They have more flexibility on how hard they push the reviewer to accept doing the specific review, or for a specific timeline, but they still get declines from some reviewers on some papers.
If peer review worked more like other publication workflows (where documents are handed across multiple teams that review them for different reasons), I think partial anonymity (e.g. rounding authors down to a citation-count number) might actually be useful.
Basically: why can't we treat peer review like the customer service gauntlet?
- Papers must pass all levels from the level they enter up to the final level, to be accepted for publication.
- Papers get triaged to the inbox of a given level based on the citation numbers of the submitter.
- Thus, papers from people with no known previous publications, go first to the level-1 reviewers, who exist purely to distinguish and filter off crankery/quackery. They're just there so that everyone else doesn't have to waste time on this. (This level is what non-academic publishing houses call the "slush pile.") However, they should be using criteria that give only false-positives [treating bad papers as good] but never false-negatives [treating good papers as bad.] The positives pass on to the level-2 ("normal") stream.
- Likewise, papers from pre-eminent authors are assumed to not often contain stupid obvious mistakes, and therefore, to avoid wasting the submitter's time and the time of reviewers in levels 1 through N-1, these papers get routed straight to final level-N reviewers. This group is mostly made up of pre-eminent authors themselves, who have the highest likelihood of catching the smallest, most esoteric fatal flaws. (However, they're still also using criteria that requires them to be extremely critical of any obvious flaws as well. They just aren't supposed to bother looking for them first, since the assumption is that they won't be there.)
- Papers from people with an average number of citations end up landing on some middle level, getting reviewed for middling-picky stuff by middling-experienced people, and then either getting bounced back for iteration at that point, or getting repeatedly handed up the chain with those editing marks pre-picked so that the reviewers on higher levels don't have to bother looking for those things and can focus on the more technically-difficult stuff. It's up to the people on the earlier levels to make the call of whether to bounce the paper back to the author for revision.
(Note that, under this model, no paper is ever rejected for publication; papers just get trapped in an infinite revision loop, under the premise that in theory, even a paper fatally-flawed in its premise could be ship-of-Theseus-ed during revision into an entirely different, non-flawed paper.)
You could compare this to a software toolchain — first your code is "reviewed" by the lexer; then by the parser; then by the macro expansion; then by any static analysis passes; then by any semantic-model transformers run by the optimizer. Your submission can fail out as invalid at any step. More advanced / low-level code (hand-written assembler) skips the earlier steps entirely, but that also means talking straight to something that expected pre-picked output and will give you very terse, annoyed-sounding, non-helpful errors if it does encounter a flaw that would have been caught earlier in the toolchain for HLL code.
You can't really blind the author names. First, the reviewers must be able to recognize if there is a conflict of interest, and second, especially for papers on experiments, you know from the experiment name who the authors would be.
>under this model, no paper is ever rejected for publication; papers just get trapped in an infinite revision loop
This could mean a viable paper never gets published. Most journals require that you only submit to one journal at a time. So if it didn’t meet criteria for whatever reason (even a bad scope fit) it would never get a chance at a better fit somewhere else).
> We are exending our previous work in [7]
or cite a few relevant papers
> This topic has been studied in [3-8]
Where 3 was published by group X, 5 by group Y, 7 by group Z and 4, 6 and 8 by group W. Anyone can guess the author of the paper is in group W.
Just looking at the citations, it's easy to guess the group of the author.
Sometimes one person is looking for an improvement in this area while someone else cares more about that other area
This is totally reasonable! (Ideally if they're contradicting each other you can escalate to create a policy that prevents future contradictions of that sort)
⸻
1. For those who care about the full messy details I have charts and graphs at https://www.dahosek.com/2024-in-reejctions-and-acceptances/
Science is about the unknown, building testable models and getting data.
Even an AI review system could help.
All the alternatives, including the ones you proposed, have their own serious downsides, which is why we kept the status quo for the past few decades.
i.e. trolls, brigades, spammers, bots, and all manner of uninformed voices.
(Actually, we already have the "open publishing" you are suggesting - it's called Blogging or social media.)
In other words, if we have open publishing, then someone like me (with zero understanding of a topic) can publish a very authentic-looking pile of nonsense with exactly the same weight as someone who, you know, has actually done some science and knows what they're talking about.
The common "solution" to this is voting - like with StackOverflow answers. But that is clearly trivial to game and would quickly become meaningless.
So human review it is - combined with the reputation that a journal brings. The author gains reputation because some reviewers (with reputation) reviewed the paper, and the journal (with reputation) accepted it.
Yes, this system is cumbersome, prone to failure, and subject to outside influences. It's not perfect. Just the best we have right now.
That's fine. I don't read eg Astral Codex Ten because I think the reputation of Substack is great. The blog can stand entirely on its own reputation (and the reputation of its author), no need for the publisher to rent out their reputation.
See also Gwern.net for a similar example.
No need for any voting.
This is less of an issue with systems where there is little monetary value attached (I don't know anyone whose mortgage is paid for by their Stack Overflow reputation). Now imagine that the future prospects of a national lab with multi-million yearly budget are tied to a system that can be (relatively easily) gamed with a Chinese or Russian bot farm for a few thousand dollars.
There are already players that are trying hard to game the current system, and it sometimes sort of works, but not quite, exactly because of how hard it is to get into the "high reputation" club (on the other hand, once you're in, you can often publish a lot of lower quality stuff just because of your reputation, so I'm not saying this is a perfect system either).
In other words, I don't think anyone reasonable is seriously against making peer review more transparent, but for better or worse, the current system (with all of its other downsides) is relatively robust to outside interference.
So, unless we (a) make "being a scientist" much more financially accessible, or (b), untangle funding from this new "open" measure of "scientific achievement", the open system would probably not be very impactful. Of course, (a) is unlikely, at least in most high-impact fields; CS was an outlier for a long time, not so much today. And (b) would mean that funding agencies would still need something else to judge your research, which would most likely still be some closed, reputation-based system.
Edit TL;DR: Describe how the open science peer-review system should be used to distribute funding among researchers while begin reasonably robust to coordinated attacks. Then we can talk :)
You could of course make it double blind, but that seems hard to enforce in practice in such an open setup, and still, hyped papers in fashionable topics would get many reviews while papers that are hardcore theoretical, in an underdog domain, etc. would get zero.
Finally, it also becomes much more difficult to handle conflicts of interest, and the system is highly vulnerable to reviewer collusion.
What makes you think so? We already have and had plenty of other ways. Eg you can see how science is done in corporations or for the military or for fun (see those old gentlemen scientists, or amateurs these days), and you can also just publish things on your own these days.
The only real function of these old fashioned journals is as gatekeepers for funding and career decisions.
Because, as an PhD who knows dozens of other PhDs in both academia and industry, and who has never heard of this magic new approach to doing science, it would be quite a surprise.
Publishing can be one part of doing science, but it's not the end-and-be-all.
And yes, I have no idea how great corporate research or military research etc are, I just brought them up as examples of research outside of academia that we can look to and perhaps learn something from.
(And I also strongly suspect research at TSMC will be very different from research at Johnson & Johnson and that's very different from how Jane Street does research. So not all corporate research is the same.)
> Because, as an PhD who knows dozens of other PhDs in both academia and industry, and who has never heard of this magic new approach to doing science, it would be quite a surprise.
And why would you expect your PhD friends to hear from that? PhD's are very much in academia, and very much embedded in academia's publish-or-perish.
At least in CS, the system can be fixed, but those in power are unable and unwilling to fix it. Authors don't want to be held accountable ("if we submit the code with the paper -- someone might find a critical bug and reject the paper!"), and reviewers are both unqualified (i.e. haven't written a line of code in 25 years) and unwilling to take on more responsibility ("I don't have the time to make sure their experiment code is fair!"). So we are left with an obviously broken system where junior PhD students review artifacts for "reproducibility" and this evaluation has no bearing whatsoever on whether a paper gets accepted. It's too easy to cook up positive results in almost any field (intentionally, or unintentionally), and we have a system with little accountability.
It's not "the best we have", it's "the best those in power will allow". Those in power do not want consequences for publishing bad research, and also don't want the reviewing load required to keep bad research out.
This is a very conspiratorial view of things. The simple and true answer is your last suggestion: doing a more thorough review takes more time than anyone has available.
Reviewers work for free. Applying the level of scrutiny you're requesting would require far more work than reviewers currently do, and maybe even something approaching the amount of work required to write the paper in the first place. The more work it takes to review an article, the less willing reviewers are to volunteer their time, and the harder it is for editors to find reviewers. The current level of scrutiny that papers get at the peer-review stage is a result of how much time reviewers can realistically volunteer.
Peer review is a very low standard. It's only an initial filter to remove the garbage and to bring papers up to some basic quality standard. The real test of a paper is whether it is cited and built upon by other scientists after publication. Many papers are published and then forgotten, or found to be flawed and not used any more.
If journals were operating on a shoestring budget, I might be able to understand why academics are expected to do peer review for free. As it is, it makes no sense whatsoever. Elsevier pulls down huge amounts of money and still manages to command free labor.
The external bias is clear to me (maybe a paper undermines something you're about to publish, for example) but I honestly can't see much additional bias in adding cash to a relationship that already exists.
This does seem true, but this forgets the downstream effects of publishing flawed papers.
Future research in this area is stymied by reviewers who insist that the flawed research already solved the problem and/or undermines the novelty of somewhat similar solutions that actually work.
Reviewers will reject your work and insist that you include the flawed research in your own evaluations, even if you’ve already pointed out the flaws. Then, when you show that the flawed paper underperforms every other system, reviewers will reject your results and ask you why they differ from the flawed paper (no matter how clearly you explain the flaws) :/
Published papers are viewed as canon by reviewers, even if they don’t work at all. It’s very difficult to change this perception.
Reviewers are not all-powerful, and they don't all share the same outlook. After all, reviewers are just scientists who have published articles in the past. If you are publishing papers, you're also reviewing papers. When you review papers, will you assume that everything that has ever passed peer review is true? Obviously not.
I don’t believe for one moment that the vast majority of papers in reputable conferences are wrong, if only for the simple reason that putting out incorrect research gives an easy layup for competing groups to write a follow-up paper that exposes the flaw.
It’s also a fallacy to state that papers aren’t reproducible without code. Yes code is important, but in most cases the core contribution of the research paper is not the code, but some set of ideas that together describe a novel way to approach the tackled problem.
In theory, the paper could work, but it would be incredibly weak (the key turned out to be either 1 or 0 -- a single bit).
There is technically academic novelty so it’s not “wrong”. It’s just not valuable for the field or science in general.
It's like a chemistry paper for a new material (think the recent semiconductor thing) not including the amounts used and the way the glassware was set up. You can probably get it to work in a few attempts, but then the result doesn't have the same properties as described, so now you're not sure if your process was wrong or if their results were.
I am not published but I have implemented a number of papers to code, it works fine (hashing, protocols and search mostly). I have also used code dumps to test something directly. I think I spend less time on code dumps, and if I fail I give up easier. That is the danger you start blaming the tools instead of how good you have understood the ideas.
I agree with you that more code should be released.. It is not a solution for good science though.
It's a bit like saying that to help reproduce the experiment, the experimental tools used to reach the conclusion should be shared too. But reproducing the experiment does not mean "having a different finger clicking on exactly the same button", it means "redoing the experiment from scratch, ideally with a _different experimental setup_ so that it mitigates the unknown systematic biases of the original setup".
I'm not saying that sharing code is always bad, you give examples of how it can be useful. But sharing code has pros and cons, and I'm surprised to see so often people not understanding that.
Also, your argument seems to be "_maybe_ they will use the exact same setup". So it already looks better than the solution where you provide the code and they _will for sure_ use the exact same setup.
And "publish the details" corresponds to explain the logic, not share the exact implementation.
Also, I'm not saying that sharing the code is bad, but I'm saying that sharing the code is not the perfect solution and people who thinks not sharing the code is very bad are usually not understanding what are the danger of sharing the code.
And I disagree with that and think that you are overestimating the gain brought by sharing the code and are underestimating the possible problems that sharing the code bring.
At CERN, there are 2 generalistic experiments, CMS and ATLAS. The policy is that people from one experiment are not allowed to talk of undergoing work with people from the other. You notice that they are officially forbidden, not "if some want to discuss, go ahead, others may choose to not discuss". Why? Because sharing these details is ruining the fact that the 2 experiments are independent. If you hear from your CMS friend that they have observed a peak at 125GeV, you are biased. Even if you are a nice guy and try to forget about it, it is too late, you are unconsciously biased: you will be drawn to check the 125GeV region and possibly notice a fluctuation as a peak while you would have not noticed otherwise.
So, no, saying "I give the code but if you want you may not look at it" is not enough, you will still de-blind the community. As soon as some people will look at the code, they will be biased: if they will try to reproduce from scratch, they will come up with an implementation that is different from the one they would have come up with without having looked at the code.
Nothing too catastrophic either. Don't get me wrong, I think that sharing the code is great, in some cases. But this picture of saying that sharing the code is very important is just misunderstanding of how science is done.
As for the other "specific data", yes, some data is better not to share too if it is not needed to reproduce the experiment and can be source of bias. The same could be said about everything else in the scientist process: why sharing the code is so important, and not sharing all the notes of each and every meetings? I think that often the person who don't understand that is a software developer, and they don't understand that the code that the scientist creates is not the science, it's not the publication, it's just the tool, the same way a pen and a piece of paper was. Software developers are paid to produce code, so code is for them the end goal. Scientists are paid to do research, and code is not the end goal.
But, as I've said, sharing the code can be useful. It can help other teams working on the same subject to reach the same level faster or to notice errors in the code. But in both case, the consequence is that these others teams are not producing independent work, and this is the price to pay. (and of course, they are layers of dependence: some publications tend to share too much, other not, but it does not mean some are very bad and others very good. Not being independent is not the end of the world. The problem is when someone considers that sharing the code is "the good thing to do" without understanding that)
It's really strange seeing how many (academic) people will talk themselves into bizarre explanations for a simple phenomenon of widespread results hacking to generate required impact numbers. Occams razor and all that.
So, no, no need to invent that academics are all part of this strange crazy evil group. Academics are debating and are being skeptical of their colleagues results all the time, which is already contradictory to your idea that the majority is motivated by frauding.
Occams razor is simply that there are some good reasons why code is not shared, going from laziness to lack of expertise on code design to the fact that code sharing is just not that important (or sometimes plainly bad) for reproducibility, no need to invent that the main reason is fraud.
Beacuse - if you'd been in academia - you'd find out that replicating papers isn't something that will allow you to keep your funding, your job and your path to next title.
And I'm not sure why did you jump to "crazy evil group" - noone is evil, everyone is following their incentives and trying to keep their jobs and secure funding. The incentives are perverse. This willing blindness against perverse incentives (which appears both in US academia and corporate world) is a repeated source of confusion for me - is the idea that people aren't always perfectly honest when protecting their jobs, career success and reputation really so foreign to you?
It's very strange to pretend that sharing the code will help the replication crisis, while the replication crisis is about INDEPENDENT REPLICATION, where the experience is redone in an independent way. Sometimes even with a totally perpendicular setup. The closer the setup, the weaker is the replication.
It feels like it's watching the finger who point at the moon: not understanding that replication does not mean "re-running the experiment and reaching the same numbers"
> noone is evil, everyone is following their incentives and trying to keep their jobs and secure funding
Sharing the code has nothing to do with the incentives. I will not loose my funding if I share the code. What you are adding on top of that, is that the scientist is dishonest and does not share because they have cheated in order to get the funding. But this is the part that does not make sense: unless they are already established enough to have enough aura to be believed without proofs, they will lose their funding because the funding is coming from peer committee that will notice that the facts don't match the conclusions.
I'm sure there are people who down-play the fraud in the scientific domain. But pretending that fraud is a good strategy for someone's career and that it is why people will fraud so massively that sharing the code is rare, this is just ignorance of the reality.
I'm sure some people fraud and don't want to share their code. But how do you explain why so many scientists don't share their code? Is that because the whole community is so riddled with cheaters? Including cheaters that happens to present conclusions that keep being proven correct when reproduced? Because yes, there are experiments that have been reproduced and confirmed and yet the code, at the time, was not shared. How do you explain that if the main reason to not share the code is to hide cheating?
I didn't care about sharing code (it's not common), but independent implementation and comparison of ML and AI algorithms with purpose of independent comparison. So I'm not sure why you're getting so hung up on the code part: majority of papers were describing trash science even in their text in effort to get published and show results.
The problem is not really "academia", it is that, in your area, the academic community is particularly poor. The problem is not really the "replication crisis", it is that, in your area, even before we reach the concept of replication crisis, the work is not even reaching the basic scientific standard.
Oh, I guess it is Occams Razor after all: "It's really strange seeing how many (academic) people will talk themselves into bizarre explanations for a simple phenomenon of widespread results hacking to generate required impact numbers". Occams Razor explanation: so many (academic) people will not talk about the malpractice because so many (academic) people work in an area where these malpractice are exceptional.
It reads as if your point is talking in circles. “Don’t blame academia when academia doesn’t police itself” is not a strong stance when they are portrayed as doing exactly that. Or, maybe more generously, you have a different definition of academia and it’s role.
I think sharing code can help because it’s part of the method. It wouldn’t be reasonable for omitting aspects of the methodology of a paper under the guise that replication should devise their own independent method. Explicitly sharing methods is the whole point of publication and sharing it is necessary for evaluating its soundness, generalizability, and limitations. izacus is right, a big part of the replication crisis is because there aren’t near as many incentives to replicating work and omitting parts of the method make this worse, not better.
And, like "scrum", "academia" is just the sum of the actors, including the paper authors. It's even more obvious that peer review is done by other paper authors: you cannot really be a paper author and blame "academia" for not doing a good peer review, because you are one of the person in charge of the peer review yourself.
As for "sharing code is part of the method", it is where I strongly disagree. Reproducibility and complete description allowing reproducibility is part of the method, but keeping enough details blinded (a balance that can be subjective) is also part of the method. So, someone can argue that sharing code is in contradiction with some part of the method. I think one of the misunderstanding is that people cannot understand that "sharing methods" does not require "sharing code".
Again, the "replication crisis" can be amplified by sharing code: people don't replicate the experiment, they just re-run it and then pretend it was replicated. Replicating the experiment means re-proving the results in an independent way, sometimes even with an orthogonal setup (that's why CMS and ATLAS at CERN are using on purpose different technologies and that they are not allowed to share their code). Using the same code is strongly biased.
As others have talked about here, sometimes it becomes impossible to replicate the results. Is it because of some error in the replication process, the data, the practioner, or is the original a sham? It's hard to deduce when there's a lot you can't chase down.
I also think you are applying an overly superficial rationalization as to why sharing code would amplify the replication issue. This is only true if people mindlessly re-run the code. The point of sharing it is so the code can be interrogated to see if there are quality issues. Your same argument could be made for sharing data; if people just blindly accept the data the replication issue would amplify. Yet we know that sharing the data is what led to uncovering some of the biggest issues in replication, and I don’t see many people defending hiding data as a contradiction in the publication process. I suspect it’s for the reasons others have already eluded to in this thread.
Also, let's not mix up "peer review" or "code sharing" and "bad publication" or "replication crisis".
I know people outside of science don't realise that, but publishing is only a very small element amongst the full science process. Scientists are talking together, exchanging all the time, at conferences, at workshops, ... This idea that a bad publication is fooling the domain experts does not correspond to reality. I can easily find a research paper mill and publish my made-up paper, but this would be 100% ignored by domain experts. Maybe one or two will have a look at the article, just in case, but it is totally wild to think that domain experts just randomly give a lot of credit to random unknown people rather than working with the groups of peers that they know well enough to know they are reliable. So, the percentage of "bad paper" is not a good metric: the percentage of bad papers is not at all representative of the percentage of bad papers that made it to the domain experts.
You seem to not understand the "replication crisis". The replication does not happens because the replicators are bad or the initial authors are cheating. There is a lot of causes, from the fact that science happens to the technology edge and that the technology edge is more tricky to reach, that the number of publications has increased a lot, that there is more and more economical interest trying to bias the system, to the stupid "publish or perish" + "publish only the good result" that everyone in the academic sector agree is stupid but exist because of non-academic people. If you publish scientifically interesting result that says "we have explored this way but found nothing", you have a lot of pressure from the non-academic people who are stupid enough to say that you have wasted money.
You seems to say "I saw a broken clock once, so it means that all clocks are broken and if you pretend it is not the case, it is just because a broken clock is still correct twice a day".
> This is only true if people mindlessly re-run the code. The point of sharing it is so the code can be interrogated to see if there are quality issues.
"Mindlessly re-running the code" is one extreme. "reviewing the code perfectly" is another one. Then there are all the scenario in the middle from "reviewing almost perfectly" to "reviewing superficially but having a false feeling of security". Something very interesting to mention is that in good practices, code review is part of software development, and yet, it does not mean that software have 0 bugs. Sure, it helps, and sharing the code will help too (I've said that already), but the question is "does it help more than the problem it may create". That's my point in this discussion: too many people here just don't understand that sharing the code create biases.
> Yet we know that sharing the data is what led to uncovering some of the biggest issues in replication,
What? What are your example of "replication crisis" where the problem "uncovered" by sharing the data? Do you mix up "replication crisis" and "fraud"? Even for "fraud", sharing the data is not really the solution, people who are caught are just being reckless and they could have easily faked their data in more subtle ways. On top of that, rerunning on the same data does not help if the conclusion is incorrect because of a statistical fluctuation in the data (at 95% confidence level, 5% of the paper can be wrong while they have 0 bugs, the data is indeed telling them that the most sensible conclusion is the one they have reached, and yet these conclusions are incorrect). On the other hand, rerunning on independent data is ALWAYS exposing a fraudster.
> and I don’t see many people defending hiding data as a contradiction in the publication process.
What do you mean? At CERN, sharing the data of your newly published paper with another collaboration is strictly forbidden. Only specific samples are allowed to be shared, after a lengthy approval procedure. But the point is that a paper should provide enough information that you don't need the data to discover if the methodology is sound or not.
I'm saying the peer review process is largely broken, both in the quality and quantity of publications. You have taken a somewhat condescending tone a couple times now to indicate you think you are talking to an audience unfamiliar with the peer review process, but you should know that the HN crowd goes far beyond professional coders. I am well aware of the peer review process, and publish and referee papers regularly.
>There are problems but they are pretty anecdotal
This makes me think you may not be familiar with the actual work in this area. It varies, but some domains show the majority (as many as 2/3rds) of studies have replication issues. The replication rates are lowest in complex systems, with 11% in biomedical being the lowest I'm aware of. Other domains have better rates, but not trivial and not anecdotal. Brian Nosek was one of the first that I'm aware of to systematically study this, but there are others. Data Colada focuses on this problem, and even they only talk about the studies that are generally (previously) highly regarded/cited. They don't even bother to raise alarms about the less consequential work they find problems with. So, no, this is not about me extrapolating from seeing "a broken clock once."
>it does not mean that software have 0 bugs
Anyone who regularly works with code knows this. But I think you're misunderstanding the intent of the code. It's not just for the referees, but the people trying to replicate it for their own purposes. As numerous people in this thread have said, replicating can be very hard. Good professors will often assign well-regarded papers to students to show them the results are often impossible to reproduce. Sharing code helps troubleshoot.
>So, the percentage of "bad paper" is not a good metric: the percentage of bad papers is not at all representative of the percentage of bad papers that made it to the domain experts.
This is a unnecessary moving of the goalposts. The thrust of the discussion is about the peer-review and publication process. Remember the title is "one of my papers got declined today" And now you seemingly admit that the publication process is broken, but it doesn't matter because experts won't be fooled. Except we have examples of Nobel laureates making mistakes with data (Daniel Kahneman), or high-caliber researchers sharing their own anecdotes (Tao and Grant) as well as fraudulent publications impacting millions of dollars of subsequent work (Alzheimers). My claim is that a good process should catch both low quality research and outright fraud. Your position is like an assembly line saying they don't have a problem when 70% of their widgets have to be thrown out because people at the end of the line can spot the bad widgets (even when they can't).
>What are your example of "replication crisis" where the problem "uncovered" by sharing the data?
Early examples would be dermatology studies for melanoma where simple bad practices were not followed, like balanced datasets. Or criminal justice studies that amplified racial biases or showed the authors didn't realize the temporal data was sorted by criminal severity. And yes, the most egregious examples are fraud, like the Dan Ariely case. That wasn't found until people went to the data source directly, rather than the researchers. But there are countless examples of p-hacking that could be found by sharing data. If your counter is that these are examples of people cheating recklessly and they could have been more careful, that doesn't make your case that the peer-review process works. It just means it's even worse.
>sharing the data of your newly published paper with another collaboration is strictly forbidden
Yup, and I'm aware of other domains that hide behind the confidentiality of their data as a way to obfuscate bad practices. But, in general, people assume sharing data is a good thing, just like sharing code should be.
>But the point is that a paper should provide enough information that you don't need the data to discover if the methodology is sound or not.
Again (this has been said before) the point in sharing is to aid in troubleshooting. Since we already said replication is hard, people need an ability to understand why the results differed. Is it because the replicator made a mistake? Shenanigans in the data? A bug in the original code? P-hacking? Is the method actually broken? Or is the method not as generalizable as the original authors led the reader to believe? Many of those are impossible to rule out unless the authors share their code and data.
You bring up CERN so consistently that I tend to believe you are looking at this problem through a straw and missing the larger context of rest of the scientific world. Yours reads as a perspective of someone inside a bubble.
Yes, sharing the code can be one way to find bugs, I've said that already. Yes, sharing the code can help bootstrap another team, I've said that already.
What people don't realize is that reproducing from scratch the algorithm is also very very efficient. First, it's arguably a very good way to find bugs: if the other team does not have the exact same number as you, you can pinpoint exactly where you have diverged. When you find the reason, in the large majority of the case, it totally passed through several code reviewer. Reading a code thinking "does it make sense" is not an easy way to find bug, because bugs are usually in place where the code of the original author looked good when read.
And secondly, there is a contradiction in saying "people will study the code intensively" and "people will go faster because they don't have to write the code".
> Remember the title is "one of my papers got declined today"
Have you even read what Tao says? He explains that he himself have rejected papers and has probably generated similar apparently paradoxical situations. His point is NOT that there is a problem with paper publication, it is that paper rejection is not such a big deal.
For the rest, you keep mixing up "peer review", "code sharing", "replication crisis", ... and because of that, your logic just make 0 sense. I say "bad paper that turns out to have errors (involuntary or not) are anecdotal" and you answer "11% of the biomedical publication have replication problem". Then when I ask you to give example where the replication crisis was avoided by sharing the data, you talk about bad papers that turns out to have errors (involuntary or not).
And, yes, I used CERN as an example because 1) I know it well, 2) if what you say is correct, how on hell CERN is not bursting with fire right now? You are pretending that sharing code or sharing data is a good idea and part of good practice. If it is true, how do you explain that CERN forbid it and still is able to generate really good papers. According to you, CERN would even be an exception where replication crisis, bad paper and peer-review problem is almost existent (and therefore I got the wrong idea). But if it is the case, how do you explain that: despite not doing what you pretend will help avoiding those, CERN does BETTER?!
But by the way, at uni, I became very good friend with a lot of people. Some of them scientists in other discipline. We regularly have this kind of discussion because it is interesting to compare our different world. The funny part is that I did not really think of how sharing the code or the data is not such a big deal after (it still can be good, but it's not "the good practice"), I realise it because another person, a chemist, mentioned it.
This is where we differ. Especially if the author shares neither the data or the code, because you can never truly be sure it's a software bug or a data anomaly or a bad method or outright fraud. So you can end up burning tremendous amounts of time investigating all those avenues. That statement (as well as others about how trivial replication is) makes me think you don't actually try to replicate anything yourself.
>there is a contradiction in saying "people will study the code intensively" and "people will go faster because they don't have to write the code".
I never said "people will go faster" because they don't have to write the code. Maybe you're confusing me with another poster. You were the one who said sharing code is worthless because people can "click on the button and you get the same result". My point, and maybe this is where we differ, is that for the ultimate goal is not to create the exact same results. The goal I'm after is to apply the methodology to something else useful. That's why we share the work. When it doesn't seem to work, I want to go back to the original work to figure out why. The way you talk about the publication process tells me you don't do very much of this. Maybe that's because of your work at CERN is limited in that regard, but when I read interesting research I want to apply it to different data that are relevant to the problems I'm trying to solve. This is the norm outside of those who aren't studying the replication crisis directly.
>I say "bad paper that turns out to have errors (involuntary or not) are anecdotal"
My answer was not conflating peer-review and code sharing and replication (although I do think they are related). My answer was to give you researchers who work in this area because their work shows it is far from anecdotal. My guess is you didn't bother to look it up because you've already made up your mind and can't be bothered.
>I ask you to give example where the replication crisis was avoided by sharing the data, you talk about bad papers that turns out to have errors
Because it's a bad question. A study that is replicated using the same data is "avoiding the replication crisis". Did you really want me to list studies that have been replicated? Go on Kaggle or Figshare or Genbank if you want example of datasets that have been used (and replicated), like CORD-19 or NIH-dbGaP or World Values Survey or any host of other datasets. You can find plenty of published studies that use that data and try to replicate them yourself.
>how on hell CERN is not bursting with fire
The referenced authors talk about how physics is generally the most replicable. This is largely because they have the most controlled experimental setups. Other domains that do much worse in terms of replicability are hampered by messier systems, ethical considerations, etc. that limit the scientific process. In the larger scheme of things, physics is more of an anomaly and not a good basis to extrapolate to the state of affairs for science as a whole. I tend to think you being in a bubble there has caused you to over-extrapolate and have too strong of a conclusion. (You should also review the HN guidelines that urge commenters to avoid using caps for emphasis)
>"sharing the code...but it's not "the good practice""
I'm not sure if you think sharing a single unsourced quip is convincing but, your anecdotal discussion aside, lots of people disagree with you and your chemist friend. Enough so that it's become a more and more common practice (and even requirement in some journals) to share data and code. Maybe that's changed since your time at uni, and probably for the better.
> Especially if the author shares neither the data or the code
What are you talking about. In this example, why do you invent they are not sharing the data? That's the whole point.
> A study that is replicated using the same data is "avoiding the replication crisis"
BULLSHIT. You can build confidence by redoing the experience with the same data, but it is just ONE PART and it is NOT ENOUGH. If there is a statistical fluctuation in the data, both studies will conclude something false.
I have of course reproduced a lot of algorithm myself, without having the code. It's not complicated, the paper explains what you need to do (and please, if your problem is that the paper does not explain, then the problem is not about sharing the code, it's about paper badly explaining).
And again, my argument is "nobody share data" (did you know that some study also shares code? Did you know that I have occasionally shared code? Because, as I've said before, it can be useful), but that "some don't share data and yet are still doing very good, both on performance, on fraud detection or on replication".
For the rest, you are just saying "my anecdotal observations are better than yours".
But meanwhile, even Terence Tao does not say what you pretend he says, so I'm sure you believe people agree with you, but it does not mean they do.
Please review and adhere to the HN guidelines before replying again.
>why do you invent they are not sharing the data?
Because you advocated that very point. You: "some data is better not to share too" The point in sharing is that I want to interrogate your data/code to see if it's biased or misrepresented or prone to error if it doesn't seem to work for the specialized problem I am trying to apply it to. When you don't share it and your problem doesn't replicate, I'm left wondering "Is it because they have something unique in their dataset that doesn't generalize to my problem?"
>BULLSHIT.
Please review and adhere to the HN guidelines before replying again.
>It's not complicated
You can make this general claim about all papers based on your individual experience? I've already explained why your personal experience is probably not generalizable across all domains.
>you are just saying "my anecdotal observations are better than yours".
No, I'm saying the systematically studied, published, and replicated studies trump your anecdotal claims. I've given you some example authors, if you have an issue with their methods, delineate the problems explicitly rather than sharing weak anecdotes.
SOME data. SOME. You've concluded, incorrectly, that I was pretending that sharing data is not useful all the time, which is not at all what I've said.
> You can make this general claim about all papers based on your individual experience?
What? Do you even understand basic logic? I'm saying that I've observed SOME paper where sharing the code did not help. I'm not saying sharing the code never help (I've said that already). I'm just saying that people usually don't understand the real cause of the problem, and invent that sharing the code will help, while in fact doing other things (for example being more precise in the explanation) will solve the problem without having to pay for the unblinding that sharing the code generate.
Sure, one reason I say that is because of my experience, even if my observations are not at all limited to one field as I've exchanged on the subject with many scientists. But another reason is that when I discuss the subject, the people who overestimate the gain of sharing the code really have difficulties to understand the disadvantages in sharing the code.
Yourself, you seems to not understand what we need for a good replication. Replication is supposed to independently demonstrate, so we build up the confidence in the conclusions. Rerunning with the same data or the same code is not enough, because it does not prove that the conclusions will remain valid if we try with other data or other implementation. When you understand that, only then you understand that sharing the code has a price to pay.
By the way, it will also explain why CERN is doing something that, according to you, has absolutely no reason to exist except for cheating. Of course, if it was the case, intellectually honest scientists would all ask CERN to cancel these policies. They don't, because there are real reasons why scientists may prefer in some case to forbid sharing code and data (not just "I don't do it myself because I'm lazy", but "I don't do it because it's a specific rule, they explicitly say it's a bad thing to do it").
And, sure, maybe it is not everywhere. But it does not matter. It's a counter-example that demonstrates that your hypothesis does not work. If your hypothesis was true, what CERN does would not be possible, it would be obviously a bad move and would be attacked.
> I've given you some example authors, if you have an issue with their methods, delineate the problems explicitly rather than sharing weak anecdotes.
These studies do not conclude that sharing the code is a good solution. None of these studies are in contradiction with what I say.
Of course, from someone who think that saying "some data is better not to share too" and conclude that it means "data is better to never be shared", or that did not understood the point of Tao, I'm sure you are convinced they say that. They just don't.
Mostly they lack critical information (missing chosen constants in equations, outright missing information on input preparation or chunks of "common knowledge algorithms"). Those that don't have measurements that outright didn't fit the reimplemented algorithms or only succeeded in their quality on the handpicked, massaged dataset of the author.
It's all worse than you can imagine.
In contrast, if the main value of a paper is a claim that they increase performance/accuracy in some task by x%, then its value can be completely dependent on whether it actually is reproduceable.
Sounds like you are complaining about the latter type of work?
If this is the case, the paper should not include a performance evaluation at all. If the paper needs a performance evaluation to prove its worth, we have every right to question the way that evaluation was conducted.
I did not dispute that peer review acts as a filter. But reviewers are not reviewing the science, they are reviewing the paper. Authors are taking advantage of this distinction.
> if only for the simple reason that putting out incorrect research gives an easy layup for competing groups to write a follow-up paper that exposes the flaw.
You can’t make a career out of exposing flaws in existing research. Finding a flaw and showing that a paper from last year had had cooked results gets you nowhere. There’s nowhere to publish “but actually, this technique doesn’t seem to work” research. There’s no way for me to prove that the ideas will NEVER work —- only that their implementation doesn’t work as well as they claimed. Authors who claim that the value is in the ideas should stick to Twitter, where they can freely dump all of their ideas without any regard for whether they will work or not.
And if you come up with another way of solving the problem that actually works, it’s much harder to convince reviewers that the problem is interesting (because the broken paper already “solved” it!)
> in most cases the core contribution of the research paper is not the code, but some set of ideas that together describe a novel way to approach the tackled problem
And this novel approach is really only useful if it outperforms existing techniques. “We won’t share the code but our technique works really well we promise” is obviously not science. There is a flood of papers with plausible techniques that look reasonable on paper and have good results, but those results do not reproduce. It’s not really possible to prove the technique “wrong”, but the burden should be on the authors to provide proof that their technique works and on reviewers to verify it.
It’s absurd to me that mathematics proofs are usually checked during peer review, but in other fields we just take everyone at their word.
It may have been for some time, but there is human social dynamics in play.
So the less brittle option obviously might be to go through all possible approaches, but this is obviously more resources demanding, plus we still have the issue of creating some synthesis of all the accumulated insights from various approaches which itself might be taken into various approaches. That’s more of a indefinitely deep spiral, under that perspective
An other perspective is to consider, what are the expected outcomes of the stakeholders maybe. A shiny academic career? An attempt to bring some enlightenment on deep cognitive patterns to the luckiest follows that have the resources at end to follow your high level intellectual gymnastic? A pursuit of ways to improve humanity condition through relevant and sound knowledge bodies? There are definitely many others.
The journal did not go out empty, and the paper did not cease to exist.
The incentives on academics reward them for publishing in exclusive journals, and the most exclusive journals - Nature, Science, Annals of Mathematics, The BMJ, Cell, The Lancet, JAMS and so on - only publish a limited number of pages in each issue. Partly because they have print editions, and partly because their limited size is why they're exclusive.
A rejection from "Science" or "Nature" doesn't mean that your paper is wrong, or that it's fraudulent, or that it's trivial - it just means you're not in the 20 most important papers out of the 50,000 published this week.
And yes, if instead of making one big splash you make two smaller splashes, you might well find neither splash is the biggest of the week.
Even though my pal did a full Gouraud shading in pure assembly using registers only (including the SP and a dummy stack segment) - absolute breakthrough back in 1997.
We did a 4 server p3 farm seeding 40mbits of outward traffic in 1999. Myself did a complete Perl-based binary stream unpacking - before protobuf was a thing. Still live handling POS terminals.
Discovered a much more effective teaching methodology which almost doubled effectiveness. Time-series compression with grammars,… And many more as we keep doing new r&d.
None of it is going to be published as papers on time (if ever), because we really don’t want to suffer this process which brings very little value afterwards for someone outside academia or even for people in academia unless they peruse PHD and similar positions.
I’m struggling to force myself to write an article on text2sql which is already checked and confirmed to contain a novel approach to RAG which works, but do I want to suffer such rejection humiliation? Not really…
It seems this paper ground is reserved for academics and mathematics in a certain ‘sectarian modus operandi’, and everyone else is a sucker. Sadly after a while the code is also lost…
You don’t have to make a ”paper” out of it, maybe make blog post or whatever if that is more your style. Maybe upload a pdf to arxiv.
Half the job in science is informing (or convincing) everyone else about what you made and why it is significant. That’s what conferences try to facilitate, but if you don’t want to do that, feel free to do the ”advertising” some other way.
Complaining about journals being selective is just a lazy excuse for not publishing anything to help others. Sure the system sucks, but then you can just publish some other way. For example, ask other people who understand your work to ”peer review” your blog posts.
Additionally, writing is the best way to properly think things through. If you can't write an article about your work then most likely you don't even understand it yet. Maybe there are critical errors in it. Maybe you'll find that you can further improve it. By researching and citing the relevant literature you put your work in perspective, how it relates to other results.
BUT... the topic is not about releasing stuff in the wild. opensource being a vehicle for research is outside the scope of present discussion. the incentives and the barrier to writing what is called an academic paper is. wild stuff does not bring impact factor, and does not get you closer to a PhD in the practical sense.
the whole paper thing is intended for sharing purposes, yet it keeps people away very successfully. its a system, not a welcoming one, that all I'm saying.
Get it included in the archives of Software Heritage and Internet Archive:
https://archive.softwareheritage.org/ https://wiki.archiveteam.org/index.php/Codearchiver
Please, could you elaborate?
The point of Terence Tao’s original post is that you just cannot think of rejection as humiliation. Rejection is not a catastrophe.
Since it is to some extent a numbers game, yes, academics (especially newer ones looking to build reputation) will submit quantity over quality. More tickets in the lottery means more chances to win.
I'm not sure though how you change this. With so many voices shouting for attention it's hard to distinguish "quality" from the noise. And what does it even mean to prioritize "quality"? Is science limited to 10 advancements per year? 100? 1000? Should useful work in niche fields be ignored simply because the fields are niche?
Is it helpful to have academics on staff for multiple years (decades?) before they reach the standard of publishing quality?
I think perhaps the root of the problem you are describing is less one of "quantity over quality" and more one of an ever-growing "industry" where participants are competing against more and more people.
In what sense? If you put it on a website, you can publish a lot more without breaking a sweat.
People who want a dead tree version can print it out on demand.
The scientists themselves are working as reviewers.
More scientists writing papers also means more scientists available for reviewing papers.
And as you say, distribution is easy, so you can do reviewing after publishing instead of doing it before.
Just like what we are doing with blog posts or web comics or novels.
Replicating research is already difficult. Finding quality research under the publish first approach will be like trying to find a needle in a haystack and I fear considerable research will be wasted on dead ends.
I don't think I ever heard anyone complain that eg Arxiv makes replicating research harder?
I wonder if it's actually optimal from the journal's selfish POV. I would expect it to want to publish articles that would be cited most widely. These should be results that are important, that is, are hubs for more potential related work, rather that impressive but self-contained results.
I wonder what the conjecture was?
Of course this is not what Terry Tao tried to do, but it was functionally indistinguishable from it to the reviewers/editors.
I actively blogged about my thesis and it somehow came up in one of those older-model plagarism detectors (this was years and years ago, it might have been just some hamfisted google search).
The (boomer) profs convened a 'panel' without my knowledge and decided I had in fact plagiarized, and informed me I was in deep doo doo. I was pretty much ready to lose my mind, my career was over, years wasted, etc.
Luckily I was buddy with a Princeton prof. who had dealt with this sort of thing and he guided me through the minefield. I came out fine, but my school never apologized.
Failure is often just temporary and might not even be real failure.