It wasn’t always that way. There were people (especially in operations) that used it as a guide for certain decisions. I was disappointed when the slogan was retired.
1,638 karma · joined August 24, 2007
The authors most likely won’t (can’t) ever know that I agreed to review their paper; it’s more of a good citizen affair. Sometimes, afterwards, it will become apparent that a particular referee was someone familiar. I would like to think that the original poster was imagining something along those lines, but your take is probably closer to the truth. Oh well.
Too many examples of “oh well, let’s try the next journal” and not enough of “gee maybe we should fix these glaring flaws”. It’s not because I’m a nice person; it’s because I would prefer the literature not to be a toxic waste dump. Nobody is entitled to be published anywhere. In an ideal world, doubly blinded review would become a part of the published record. At some journals it already does.
The reviews offer valuable context, which is often sorely lacking in high profile venues. (The canonical examples of STAP and arseniclife come to mind, but also much more subtle details where an overall sound paper somehow only gets cited for the one shaky assertion in the results)
JMHO
There’s also an ulterior motive for some lines of work, where sending a paper to be critically reviewed by rivals is a mechanism to torture test the work & conclusions.
This can be taken too far (especially in biology, the glam fluffing and $million additional irrelevant experiments are legendary), but in principle, if even your most motivated critics can’t find a fatal flaw in your work, it’s a reasonable bet that the work is sound. At least, that’s the principle. Editorial overrides sometimes break this safeguard, though (lord knows I’ve seen a few).
You can always put work up on arXiv if the goal is priority or distribution. Peer review is, in principle, designed to ensure that work is new and true (and that what is new is true). It’s not meant to fix flawed work, but to make good work better.
Two billion people living in an autocracy ... if Google still believed in Don’t Be Evil, this would not be acceptable.
Didn’t Pakistan advertise a /28 at one point that blackholed YouTube?
Good times. (/s)
It attacks a hard, hard problem and I’m super happy that it will eventually replace Mathematica so that anyone who wishes can see how the black box works.
It doesn't take that long to count citations, which defuses the usual excuse for using impact factor as a metric. (And some really shitty journals have surprisingly high impact factors -- they just don't publish much of anything in a given year, e.g. Science Immunology has exactly one primary research article in its entire August 2018 issue!)
Ultimately, it's funders who decide what "counts", because indirects from grants (money money money) is the only real currency. You can publish once every five years in the Journal of Irreproducible Results, but if you have a $20M P01 for those five years, you're getting promoted anyways. So when Chan-Zuckerberg or the Gates foundation says "OA or nothing", then it matters. Universities will keep licking their balls until the purse strings tighten, but once that happens, hoo-eee! It's amazing how quick them OA pubs count for promotion. (Secret: it was never the publications.)
Not that much. Nature sends me paper versions of most of their journals for free (so does Blood, so do some others) for (as far as I can tell) the sole purpose of getting me to read the ads. It's fine, I suppose, but I preprint all my papers anyways (some go to Nature-brand journals, most don't), and honestly their editing isn't even playing the same sport as, say, NEJM (which really does add a lot).
Most of the time the versions on *Xiv (not just my papers, but most other authors' too) are at least as good as the ones that get published in vanity journals. Sometimes they're better, since there are times that a topic really is complicated enough to take 50 pages and 30 figures. Rarely, but still -- it's not like the same 30 figures aren't just crammed into "panels" (smaller, harder to read figures) in glamor journals (or tossed into the supplement; https://www.ncbi.nlm.nih.gov/core/lw/2.0/html/tileshop_pmc/t... is one of my favorite figures of all time and it's stuffed into an "Extended Data Figure" FFS!).
Chan-Zuckerberg and the Simons Foundation pay for bioRxiv and arXiv respectively, seemingly without much trouble, so I contend that publishing papers isn't actually that expensive, certainly relative to the benefit of doing so in a way that is accessible to the public who funds most of them.
Reading tarted-up flag-planting arXiv papers can be tough going even for practicioners, to say nothing of the general public. But ultimately the general public will have to vote on propositions, or vote in representatives, to pass sensible (not hysterical) regulations.
For example, after an Uber lidar-equipped vehicle killed a Phoenix jaywalker (a separate part of law that I do not like, but acknowledges physics and shitty drivers), Uber halted testing in most markets. This is sensible based on public reaction, but three major problems compounded to take Elaine Herzberg’s life:
1) the lidar did not function as designed, 2) the backup driver did not function as instructed, and 3) Ms Herzberg did not obey the law.
This is a situation where the facts eventually became clear, and still it is difficult to say that one side or the other overreacted (personally I think Uber did the right thing PR-wise, but I also worry that less testing means slower progress towards safer cars, since human drivers are legendarily shitty and much less safe than conservative self-driving cars with functional lidar).
Now let’s consider more complicated problems, such as replacing medicinal chemists with reinforcement learning systems, or China’s AI approach to “social credits”. Both have potential benefits and huge potential drawbacks, and both are relatively easy to grasp compared to some thornier cases. One is purposely obscured due to a dictatorial regime; one is obscured not by administrative fiat but by copyright law and a very silly tradition in academic and quasi-academic research. Hiding results in paywalled “prestige” journals rather than making it available to the general public is seen as a mark of distinction (and avoiding a $30k-$50k open access charge is just sensible conservation of research funds, in this scenario. I know because I am an academic and I personally think OA charges are a waste, though for different reasons.)
Anyone who wants to [break the law] can obtain the primary documents in the former case via sci-hub. That’s just a fact. But is this any more reasonable than Georgia allowing companies to charge for access to state laws? Is it reasonable for people to pay twice, in most cases, first for the research* and then for the report, when faster dissemination of primary sources can directly inform debate over some important issues?
It seems silly that we would have a system where others are expected to pay for, digest, and opine upon research done by the primary authors. It would not be necessary if the norm was to disseminate AI breakthroughs in a format like distill.pub, where the whole point is to explain both the results and their context. But until academic and quasi-academic researchers can kick the “prestige” habit, that’s unlikely to happen, and the public will be left to listen to pundits (often idiots) rather than forming their own well-informed opinions, and that sucks.
No time to edit, but hopefully you get the idea. Disclaimer: I am an academic, applying “AI” to medical care for children with rare and deadly diseases; I used to work at GOOG a million years ago; and what I see among the worst hucksters sickens me, because I think it’s going to kill people in spite of better competing ideas that could improve the human condition. I’m just one person with one vote. I’d like everyone else to form their own opinions, because theay might change mine. Thus primary sources are critical.
* anyone who does not believe that taxpayers subsidize industrial & corporate r&d via tax credits and incentives is quite naive; it is not different from academic competition in that regard.
It’s not like CS researchers are the ones turning science into a dumpster fire to sustain 30% profit margins.
Most of the more hysterical screeching about AI, automation, etc is reminiscent of buggy whip manufacturers bitching about cars.
Nature and Science are less bad than, say, many Elsevier imprints. In most cases I feel like Nature or its daughter journal genuinely improved the manuscripts we sent them, at least the ones they published (and some they did not!). But this is also true of eLife and Genome Research. NEJM is wonderful to work with as an author, but in retrospect I sometimes wonder if they were too nice to us (Stockholm syndrome?) and have had this thought about some well-regarded specialty journals. In such cases, it is hard for the news staff to justify reporting on the underbelly.
Not to put too fine a point on it, but I had a paper last year in a journal with an impact factor of 15* that was accepted in 15 minutes. It took longer to sign the copyright transfer agreements than to "review" the work. (I have the timestamps on the emails to prove it, although I'm not going to do that because one of my coauthors is an editor.) There's just no way that peer review was adding any value in that case. And yes, it has already been cited by other groups.
* Yes, IF is a silly metric. But the parallel is just too good to pass up. One minute per point of impact, I suppose.
Mathiness is something that can usually be detected by reverse engineering the work and seeing if a derivation was important. Unfortunately, I learned this in my own discipline (statistics) by comparing how implementations actually functioned to how they were originally derived and presented. My own dissertation contained a rather terrifying bit of analytic derivation that, upon finding suitably large samples, yielded to an approximation that was many orders of magnitude faster. But hey, my committee loved that gamma.
The issue of reviewer scarcity is of a piece with all serious research. If it was easy, somebody would have solved it already. Post-publication review is the only sustainable model, and it's more beneficial for the reader if they go in with a skeptical mindset anyhow. Journals and conferences simply cannot serve as more than the weakest of filters.
“You know what we need? More funding for shitty underpowered studies in immunocompromised mice, and less funding for analysis & methods work. Besides, if they’re any good, AmaGooFaceSoft will just steal them anyways.”
Ha ha only serious. You are 1000% correct, and I can only hope that NIH will one day discover the same thing. Instead of destroying working APIs.
It was a necessary perspective check
I hear there’s some sort of intertubes system to more efficiently disseminate papers for review & reading nowadays
And in about half of those instances, it simply reminds me why I love not relying on the “goodwill” of competitors.
Rigor: Feller vol 1, Casella & Berger 2e
I.e. expose the underlying low rank of nearly any huge sparse data matrix with an SVD or NMF. Enable fast recovery with a shitty (CS-wise) hash function. Recover most of the information about an observation's neighbors in a fraction of the time taken by many other approaches.
What's popular for ML benchmarking these days is not necessarily the same as what's needed for a specific application. It's a useful proof to keep in mind before prematurely optimizing with overly complicated approaches.
When Cover & Hart proved that the error for k-NN classification is no worse than twice the Bayes (optimal) error, "machine learning" as a phrase had not yet been observed in the wild.
http://ieeexplore.ieee.org/document/1053964/
EE, CS, stats -- these are your fundamentals...
Aka do you work for Epic
EHRs are a farce. They've made a few people very wealthy at the cost of widespread misery for patients and doctors.