The Sci-Hub Effect: Sci-Hub downloads lead to more article citations

My uncharitable read is they slapped an interesting yet unsupported conclusion onto a rather boring observation.

Edit: In particular, it seems to me that papers in their control group are available on SciHub (correct me if I’m wrong), just nobody bothered to download them in the time window they analyzed. Which directly contradicts the “limited access” part of their claim.

One thing I would do a lot with Google Scholar is find a highly-cited, somewhat recent (past decade or so) paper and then click into its citations to find the very most recent related work. IIRC this interface was chronological so there was no big-journal bias and I found a lot of stuff in random journals. Low-friction quality-filtering through SciHub was very helpful for me in this process.

I worked in a theoretical field, so it's actually possible to assess the quality of a paper just by reading it. If you're doing something experimental you may have to lean on the big-journal filter a little bit more.

Under what conditions do you see that changing? You yourself say how useful it is, but discount the effect it has on the citation graph. Could we say the samething about arxiv?

This is exactly why research needs to be open and freely available. The esoteric, rarely cited papers are locked behind paywalls and away from search engines. And in doing so, their impact fades over time as the flock centers around a single vein of focus. The future should look like a net not a river.

I am not an academic and I read papers for their metalessons in areas that interest me. But without Sci-Hub, lots of research would not be accessible outside of its niche, esp if it was unlucky enough to not be in the right journal at the right time.

On hilarious pattern I see, is when a bigshot in a field publishes a new paper, there is a race to cite it by some nobody. Basically the academic publishing version of "first post".

This one managed to get 700 upvotes on HN and still be meaningless.

Thanks for writing. I've taken a quick look at your paper and its OSF page, and you all clearly know what you're doing and have some great work here. I am not your target audience (I am not an academic), but if I were peer reviewing this, I'd suggest one of two avenues.

1) As Angrist and Krueger put it, "In our view, good instruments often come from detailed knowledge of the economic mechanism and institutions determining the regressor of interest...progress comes from detailed institutional knowledge and the careful investigation and quantification of the forces at work in a particular setting." With this in mind, is it possible to contact the maintainer of Sci-Hub, email her your paper, and ask if she can think of any plausibly exogenous sources of variation in papers' availability on Sci-Hub? Some possibilities that come to mind for me:

* In my own experience, papers without DOIs are often harder to find on Sci-Hub. Are there closed-access journals that don't mint DOIs on which you could use propensity score matching to create something of an apples-to-apples comparison?

* Is there an identification strategy somewhere in journals with optional APCs for which only some of the papers are open access?

* Were there ever any sci-hub outages that lasted periods of many months, and, however long down the line, did the citations of paywalled articles decline relative to their open access peers?

2) If none of this is possible, I suggest removing all causal language from your article and sticking to a predictive model. Observational research is appropriate for exploring and identifying causal hypotheses rather than confirming them; no statistical techniques that attempt to control for unobserved population heterogeneity will persuade me otherwise (I am admittedly an extremist on this position, but my own views hew closely to those of Gerber, Green and Kaplan (2003): http://www.donaldgreen.com/wp-content/uploads/2015/09/Gerber...). The Lewbel (2012) paper you cite looks interesting but, to paraphrase Gerber et al., statistical techniques can't account for nonstatistical sources of uncertainty (the inherit unnkowability of whether you've specified the 'correct' model, in the absence of randomization or exogenous variation, means we can't say anything about the biasedness of your estimation procedure).

Observational research is great! Generating hypotheses is as important as confirming them. It's just that some research designs license causal inference, and some do not.

Best of luck.

Without the proper statistical analysis it just becomes a paper on how highly cited papers get downloaded from Sci-Hub which isn't very interesting.

Note the words used in the argument: "could have a common cause", "without necessarily being a cause".

The logic that fails is the one that depends on a common cause being impossible and concludes that one thing is necessarily causing the other.

(It plausably may do so to some extent, but that paper doesn't prove that nor gives a good estimate of the effect.)

On the matter of Sci-Hub. I already have access to all the journals I commonly use via my university. Those journals almost certainly track downloads (Elsevier even has a recommender system when you download a paper). All Sci-Hub does is reduce friction for researchers who can't be bothered to use their institutional login while off campus. Admittedly that includes a lot of people, but for the people who measurably contribute to citation metrics, does Sci-Hub actually improve availability? I doubt it, at least in universities with budgets for subscriptions. Before Sci-Hub there were always routes to get papers: ask the author or ask a colleague in a neighbouring university if they have access. Or you can ask your library who may be able to get it.

My point is that generally it was a bit more work sometimes, but in the grand scheme of publishing a paper, it wasn't so bad to network to get access to things.

It's certainly enabled the general public to access research and no doubt tons of small companies are using it to avoid paying for subscription fees, but those people aren't significantly publishing papers in places where we can measure it.

All this study shows is that paper downloads from anywhere is a proxy for popularity/hype which can be a proxy for citation count. I'm sure Elsevier or Taylor or Nature could provide a similar correlation between downloads and citations. There is also a bias here - if I'm looking for papers, I'm probably going to download the most highly cited stuff first because that's a weak signal that it's a useful paper.

I'd be more interested to see, as you said, temporal analysis of before/after Sci-Hub.

Or alternatively geographical studies - does this affect lower income countries more? I imagine that this is a boon to researchers who didn't have access because their institution couldn't afford it.

[1] https://www.elsevier.com/connect/citation-metrics-and-open-a...

[1] https://en.wikipedia.org/wiki/Open_access_in_the_Netherlands...

1. Searching for the title often gives you a PDF in the first few results already. Authors usually have the right to upload a version on their personal website or arXiv.

2. There is Google Scholar [1], which has links to PDFs on the right of the result (if it knows one). It is better at this than regular Google search, in my experience.

3. Manually searching on the author's website (or former website, if they moved) sometimes proves successful, although this is relatively more effort.

4. As mentioned, writing an e-mail to the authors works as well. (If you buy a paper, the authors get nothing, so there is no incentive not to share it.)

[1] https://scholar.google.com/

That's encouraging to hear from at least someone since it doesn't meet my experience. What I rather see in the few fields I still care about is that we're flooded with a mass of unoriginal and uninspired papers, many using a ML approach, where the purpose is clearly to get graduation or tenure rather than advancing the state of the art. It's happening to a degree that even assessing the major contributions in a field and separating me-too publication from the few original and foundational works has become impossible, similar to how general web search has become pointless. I'm all for free access, but 1. major works have always been published as author's copies with free public access 2. I really don't see any advancement in scientific quality at all as academic achievements are becoming just stepping stones and academic institutions career networks more than anything else.

Edit: also want to mention citeseer as my search engine of choice which seems to have improved a lot after their rewrite ten years ago (which made it useless for me)