Bad stuff going down at the American Sociological Association
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
This is a highly opinionated article against recent ASA opposition to a law that would require them to release taxpayer funded research publications for free immediately. Currently there is a 12 month delay before they need to release which allows them to generate revenue via subscriptions. The ASA says this is bad because it removes the source of funding for peer review so there will be negative effects on academic integrity.
The author frames this as opposing open science and they allude to other integrity problems in academia such as plagiarism and cheating.
My opinion: Kinda seems like it boils down to a conflict between wanting an open and centralized system. I cant fault the ASA on one hand because this does fundamentally weaken them, but on the other hand maybe the best system doesn't include a central institution responsible for shaping the field. But thats not currently the case, and the ASA exists, so it seems like they're acting rationally.
It's not the ASA taking the public funding. The folks who do the research should make their research results available for free, irrespective of what the ASA wants to do: the value of an ASA or any other similar association is in "being a peer-reviewed publication platform" that publishes vetted research (well, ideally. We all know the problems here). For that, having a publication publishing right where you are the only periodical that gets to release a product consisting of collections of research results does not interfere with any law that says that if you, as researcher(s), take public funding to do research, your research data and results should be freely available.
Dont come back and whine about it later.
> Despite a petition signed by many ASA members, and a resolution from its own Committee on Publications “to express opposition to the decision by the ASA to sign the December 18, 2019 letter” — which the ASA leadership never even publicly acknowledged — ASA has not uttered a word to alter its anachronistic and unpopular position.
This isn't a "centralized vs. open", it's "rent extracting journals vs. scientists"
The whole crux of the argument is that this is research that is federally funded
Whether or not a cartel is “rational” completely misses the point. It’s publicly funded research so it should be public data. Full stop no caveats or attempts to maintain the current system.
Never.
Reviewers and paper judges don’t get paid.
Arxiv and OR have done more for researchers than the scientific publishing cartel ever has
The field of ML moves as quickly as it does because doing closed access ML separates you from the community
Many of the researchers are absolutely fine with this rule. It's only the top ASA leadership and the rent extracting journals that have a problem with no longer facilitating journal's rent extraction from publicly funded research.
While I'm all for open access (so long as authors don't need to pay to publish, which is often the case with open access journals), it would be a mistake to think research is publicly funded in the US to the same extent that it is in Europe or Latin America.
This can also be a form of money laundering, though. You can easily end up completely government funded, but only through indirect means.
What I'm afraid of occurring is over-leveraging classification of research as a means around this, if for no other reason than it's easier to implement/publish.
...of course they do.
Some of this depends on who you think the readership will be. Those scientific societies provide a sort of floor on the quality of what they publish, which allows non-experts like me to read the papers with some level of trust in the results.
Or, a layer of indirection away, it gives science journalists a source of stuff to write about that's at least somewhat validated. I do worry a bit about what happens to science journalism, which is already generally of poor quality, when they start reading random junk on arXiv.
Of course, there are alternatives where the papers could still be free. The government could fund the journals, arXiv could implement do their own peer review system with quality indicators. Probably more I'm not thinking of off the top of my head.
I have no first-hand experience, but my intuiting would be that a lot of the gatekeeping these societies do is focused on reputations, organizational hierarchies, and funding considerations. Which I would not really consider productive, especially in light of the abysmal state of reproducibility and the apparent lack of incentives to tighten things up.
I don't think "cannot" should be the standard. If the same or better standard of peer review isn't provided (empirically) by some other system, then that's a legitimate criticism of the other system.
There are certainly other ways of getting to the same thing, but what I think a peer review process in isolation lacks is a sense of trustworthiness of the peer review.
I.e. as a layperson, I'm not particularly qualified to gauge the quality of an arbitrary paper. I am implicitly not qualified to gauge the quality of peer reviews, because I can't gauge the paper itself.
I can defer to the opinion of experts, though. These societies unify the diffuse expertise of who knows how many people. I.e. I don't work in sociology, so I have no idea who is a reputable sociologist, but I have some degree of confidence that the peer reviewers they choose aren't total quacks.
It's a useful heuristic for non-experts. I don't know anyone or anything in rocket science, but I can defer to what NASA says. I'm not a marine biologist, but I can defer to the NOAA. Etc, etc. There's value in that heuristic, even if it's not perfect.
> I have no first-hand experience, but my intuiting would be that a lot of the gatekeeping these societies do is focused on reputations, organizational hierarchies, and funding considerations.
This is indirectly useful to my thinking. The society is focused on funding and income, which is largely predicated on them being trustworthy (theoretically), which gives them an incentive to provide quality peer reviews. I would imagine a society with 0 standards loses subscribers pretty quickly.
arXiv could certainly do the same thing, but my impression is that arXiv is more focused on making papers available than being an authoritative entity on anything. "This paper is on arXiv" doesn't imply the same level of trust that being in a journal does. They derive value from being open rather than from being trustworthy.
arXiv definitely has value as well, I don't mean to imply that it doesn't or is bad in some way. It just has different goals, from my limited knowledge. It's likely invaluable to people in their field, especially for those qualified to gauge the trustworthiness of an individual study.
arXiv could definitely build some kind of trust system like that if they were interested, but I really think their primary audience is experts who don't have a need for a trustworthiness score simple enough for a layman.
That's not totally obvious without sorting out the details. Similar to copyright, it seems to be inefficient (in the sense that it artificially restricts supply) but certainly a lot of people think it helps improve the overall supply of creative works.
That's because it is--more precisely, it's opposing open taxpayer-funded science. We the taxpayers already paid for the research once. Then we have to pay again to read the papers sooner than 12 months after publication?
I don’t deal with the ASA much, but I do use the PNAS a lot — their HTML paper layout isn’t bad and they do maintain a nice archive. The National Academy of Science deserves some money for good web work, I just don’t know whether it’s just for them to take it from the embargo.
Also, how much marginal utility does "the public" gain when you reduce or remove an embargo?
But what is the utility that they provide?
They are no longer needed for distribution. They have never been able to validate works (despite the public perception thinking that's what they do). Scale makes them unable to effectively weed out low quality work (process is always noisy, scale increases noise). As far as I can tell, they only provide a metric that can be used by governments and academic institutions to evaluate their employees, but this gets noisier and noiser by the year. It also incentivizes "novel work" but work that isn't too novel because highly novel methods are often not effective at first despite being worth exploration. They also discourage replication studies, which is the cornerstone of science, as well as surveys, opinion papers, and more.
I really just do not see what they offer to us as researchers. They don't pay us to review. That time either comes from the academic institutions, gov money, or generally "free time". They don't pay area chairs, metareviewers, or others. They charge $30-$200 for a paper, on a shitty difficult to navigate website, where many open platforms like arxiv, openreview, and various *xiv host these works for free and are quite searchable with open APIs and many tools for easier navigation.
So what do they do? Because it looks just like a system that is really good at extracting money from academia and governments.
> So what.
100% this. There are far too many things we just do because. There are far too many things where we actively create "harm" by not changing. Where that harm is more by continuing than were we to change. Momentum is a hell of a drug and lobbyists only make this more problematic. You're right that they are rationally acting in their own interest. But we have to consider the difference between public interest and business interest. That's the whole fucking point of a government, is to protect public interests from individual interests. Where is our fucking social contract?
I'd lump it in the same category as the other things mentioned earlier: being hypocrites.
Like religion preaching one thing, practicing another. Or politicians being for the people, yet willing to enact policies or stir up wars resulting in the death of their people.
Academia is supposed to be about learning, knowledge, and growth, but locking down research and not calling out cheating and plagiarism speaks to the same rot; many at the top are just as selfish and negligent as the worst politicians and religious leaders.
But they sit at the top of their ivory towers and since we assign sacrosanct status to academic institutions (at least for the most part), they are never accountable.
This statement is normative and has no relation to how laws are actually created.
Can you name a framework of laws that was implemented by a sovereign, explicitly developed to resolve the problem of heterogeneous goal vectors for a defined political population?
I’m unaware of one.
All foundational “constitutions” I’m aware of were written - not as treaties that were negotiated as partners during peace that have explicit and well defined ethical principles that need to be accommodated for - rather each side claim discrete and finite sovereignty and only at the point that one group shows overwhelming violent dominance is imminent do groups grudgingly carve out their own territory.
So no, there is no coherent process for lawmaking that does anything other than encode the desires of the dominant political arm into the structures of its own making.
More broadly, "resolve etc." in the above quote sounds like a mathy expansion of "compromise", which happens in almost every legislative body every day. Although maybe that's done implicitly and not explicitly.
Academics need funding, funding allows them to keep going. Institutions that academics work for have bloated overheads typically going to admins who are tiered in such a way that there is just a single admin at every level.
As the overheads go up, the pressure to publish and win awards does too. Most awards for public research say you can take up to 7% of the award as unobligated funds. The result is that institutions want higher operating costs, to push the bottom line and increase their 7% share.
Fact: peer reviewers are mostly unpaid. Journals might have some paid staff, but the way peer review works is academics are expected to do the actual reviewing work for free. Doing this a lot and being famous in your field gets you a fancy title like "editor", but that doesn't usually come with money attached.
Peer review itself would work just as well in an arXiv + easyChair model like some of STEM does it, replicating the value of journals for a much lower cost (and saving some trees in the process).
Public access to publicly funded research is long overdue.
RIP Aaron Schwarz... he wasn't wrong [in his attempts to share publicly-funded research], just early.
BUT I personally support lib-gen, so I absolutely do not respect the concept of "Intellectual Property." I have contracted for several opensource projects which still find ways to become profitable (usually by distributing code with hardware).
"Privatize profits, subsidize losses" == take taxpayer funds, charge taxpayers for access IMHO
I think it's perfectly fine to selecting to work on open source projects based a principle of not respecting intellectual property rights. But it's another think to flaunt those laws and not be expected to be held accountable. What you can't expect IMO is think your lack of respect for a law somehow means you're above it. (That's not saying the govt didn't overreach in the Schwartz case; I'm not familiar enough with all the details to form a strong opinion.)
Thank you for this discussion in fostering better understandings [e.g. your perspective].
----
One of my sick optimisms about generative AI being trained upon all this human creation, is that it may actually lead to the disolution of IP ownership... one can dream =P
PyratB
I've been advocating for a model where we publish to OpenReview.
It supports revisions, comments, links to GitHub, link to community implementations, tags/keywords, data linking, and more. It seems like the most complete platform for migrating to a system of true peer review. Where peer review actually refers to review from peers and not "3-5 random people in your field incentivized to reject your work adversarially complaining about your work who may or may not know anything about your subdomain." Venue reviewing is a joke and our methods don't scale, as they depend entirely on good faith efforts and consideration of nuance. People see service and reviewing as a chore and there's little to no incentive to provide quality reviews.
There are still things I think need work to make this model successful. I think it is a bit silly that we require our communication system to be printed on a piece of paper. We should have ways to embed media like videos and demonstrations (even if it is an API call to hugging face). But the current system feels like we're still in the 1700's with a sprinkling of 21st century because now we don't have to hand draw figures.
How anybody not associated with the cartels could even tacitly support that is mind-boggling
Obviously making science available for everyone is good, and the ASA has no argument.
But the article minimizes the fundamental problems in academia.
> 2. Academia is more open than much of business, government, and organized religion, so it’s easier for us to see the problems.
That's not even remotely true. Academia is more closed, more obscure, more small network, with fewer sunshine laws, than any business, government or organized religion. It's far harder for us to see the real problems.
> 3. So much of the enabling of cheating in academia just seems so pointless. It’s not cool when companies pollute, but, hey, you can see the reason$ they’ll want to do so. But what does the American Sociology Association get out of fighting against open science, what does the University of California get out of tolerating research misconduct, what do the American Statistical Association and American Political Science Association get out of giving rewards for plagiarists? Nothing. That’s what so damn pitiful.
Cheating in academia has great rewards! The ASA gets money. UC gets prestige, donors, etc.
The fundamental mistake the author makes is that academia is not one thing. There are academics as we've always known then and then there is the administration class.
The people who do the work mostly want to get to the truth and get things done. The administration is into face-saving, donations, endowments, insanely expensive buildings, adding another VP of something, yet another layer of paperwork for every action, etc. the truth and what's right isn't aligned at all with the administrative class.
I peer review because it is what I do: I am a scientist. My university takes it into account while conducting merit reviews, but whatever extra I earn through merit pay raises, is paid for by my University and NIMH grants.
The journals are complaining that my donated labour won't be as valuable to them anymore, even as it becomes more valuable to the scientific community.
Some of these academic preprint/archive sites are experimenting with "a place for reviews".
There's a couple of issues (I hesitate to say "problem" but it could be seen that way):
1. How do you choose who does reviews?
2. Why should the archive service host those reviews?
Currently, in general, an editor asks reviewers to review, they review, and it's part of the editorial process leading to publication. This is a sort of "gatekeeping" model, for better or worse.
If you let reviews into an archive service, how do you do that? If you let everyone do it, that's fine, but then do you need arXiv etc to host those reviews? Should arXiv etc host those reviews? If you do have reviewers selected, how do you do that?
I'm not saying there aren't answers to these questions, but it does get a little complicated. Pay-to-publish has its own huge problems imho for example.
In general, academics don't know what to do to mark value in the absence of a "stamp of approval" from an "endorsed expert". I think that's what's underlying all of this.
In terms of whether reviews should be public, there are pros and cons to anonymous review. The cons are easy to see. When I read a paper I don't know whether the reviewers asked a question and gotten a response about a concern I have about the paper. Often clarifying results in response to a reviewer's inquiry can be shared in the letter but never make it into the paper. I always ask for a brief sentence describing those additional checks to be put into the paper briefly or in supplemental, but that is me.
The pros of anonymous unpublished reviews are slightly more nuanced. Sometimes stupid but innocent mistakes happen, and these can be embarrassing. Part of the value of review is having the chance to have fresh eyes on it, but doing so in public view can push people to get grounded in their position as a matter of reputation. Another issue is that criticizing a paper from a well known research is a bit harder when it will be published, and especially when it isn't anonymous.
I'm all for publicly-funded research being publicly available; but I have never regared sociology as a rigorous discipline, so in this case I don't care very much.
(I think the site is hugged to death)
Isn't this a significantly bigger issue because of how much these various studies are used to drive public policy that affects everyone?
https://www.asanet.org/about/governance-and-leadership/audit...
Possibly, but given the organization, I'm not sure it could be replicated
APA and friends have been in civil war for like 15 years. Certainly not 2019, but yes they oppose open science. Surely it has to come to an end at some point?
Really started in 2005 when you must declare your biases. Sociology has never been the same. You can basically ignore any sociology pre-2005, and oh look some classic names like Jung as coming back?
But holy crap has the replication crisis slapped sociology in the face.
The minor update to the dsm5 and the expected major delay of the dsm6 will be huge. Practically an entire rewrite at this point is needed.
I expect there will be a new diagnosis called 'neurodivergent' which will be defined as 'not bad, but different'
Autism, ADHD, OCD, chronic pain/ fatigue/ somatic disorders, tics, and bipolar
These will all be unified under the same title as they are all the same thing.
LGBT, Dyslexia, Dysgraphia, Dyscalculia, Crohns, IBS, Colitis, Rheumatoid, osteoarthritis, Celiac, sjogrens, addisons, guillain-barre, Dental inflammations, Diabetes? Chronic pancreatitis, Herpes or mouth sores
All going to show up on the ND spectrum.
cPTSD will need to exist but perhaps just be neurodivergent as well.
There's going to be a major overhaul of the personality disorders. Trauma informed.
etc etc.
Then again I expect I'll be flagged for violating the obvious rule:
"All truth passes through three stages: First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as self-evident."
This is a joke. Public funding ought to equal public results. Both to the claims, the paper, the raw data and the peer reviews.
Having said that; peer reviewers and disproving older papers should earn more money than making claims that cannot be reproduced.
Also the practice of ranking scientists and universities by citations is moronic. I could in theory write a paper that claims the earth is square. Now a 1,000 friends and colleagues quote me, explaining I don’t make sense, but somehow both I and my university will RISE in ranking?!
Beware. It's a common trick on forums not entirely unlike this one.
But maybe there are other factors that make it a net negative? (I haven't really followed academia since concluding my studies.)
One of my biggest gripes with academia. More and more are universities and those who run it act like private corporations than non-profit research institutions
So, other countries and entities can fund weapons, armies, spies, but a paywall is a bridge too far to cross? That’s an extremely weak argument.
It does not cost $1k a year to have access to these publications as an individual, anything our "enemies" would want could be obtained for effectively zero dollars.
Generally speaking, science (and particularly publicly funded science) is an internationally collaborative process. Anything that would be of material harm in the hands of a rival government should either be classified or kept well protected within the bowels of private industry.
Is there any reason the believe that science publisher's gatekeeping has ever been a significant roadblock to nation-states that are already capable of industrial espionage and intelligence activity?
Peer reviewers usually don’t even get paid at all, and getting funding for trying to replicate results is often non-existent, meaning at best it gets done by a couple of undergrads