Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals".
"I feel like you should be able (forced even!) to cite things that aren't journal articles- even blog posts- if they contributed to your body of work. But citing things that aren't prestigious reduces the perceived prestige of your own, which is a perverse incentive not to cite."
EDIT: I can't seem to figure out how to link to specific tweets, so this is the guy that said it: https://twitter.com/_delta_zero
He has a great point. If the proof is correct, you should cite the raw document. If you have to cite it through a proxy scholar, it feels like a form of gate keeping.
Here's the tweet you referenced: https://twitter.com/_delta_zero/status/1054820799473836032
This only really matters on the receiving end of the citation. I.e., if lots of people are citing your blog or forum post, then it can be a good idea to take your content and put it in arxiv or similar. Just so that the citations "count" for you in the academic setting.
And similarly, it's a bit rude (or at least odd) to cite someone's blog post when there's an accompanying published paper that describes the same work.
But citing informal sources where no formal source exists is not (and afaik never has been) a problem.
Perhaps this is better seen as a substantial defect in the fact that the academic web-of-trust model implemented in publication is based on quality of source (e.g. journal's prestige, or the submitting scholar's prestige) rather than quality of scholarship (e.g. whether a critical mass of researchers, rather than 1-3 anonymous reviewers, find it compelling and 'sign off' on it being good work).
I hope (and suspect) that whatever emerges or is developed to replace the paid journal model of academic trust is designed to handle these situations gracefully.
For wikipedia, you should reference the specific version of the article, which will never change.
For blogs, or wikipedia articles you feel might get deleted, you can reference a specific version from archive.org.
In fact, I fear that the trend to publish supplemental material may lead to another crisis because a lot of that material is not managed and archived well and may disappear without warning. Some papers contain important details like long, involved and non-obvious calculations only in their supplements and are not really reproducible without them.
You see very similar things happen from time to time where there are "folklore" theorems in mathematics that someone eventually writes down 20 years later. Sometimes they're able to give an attribution, and sometimes not. But always they give a formal proof using modern notation and credit the idea with whatever fidelity they can.
https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUx...
> I used to think it was because as a quasiscientific community we highly value peer review but ML people are totally fine with citing papers on arxiv that haven't been accepted anywhere as long as you can replicate the results - I think perceived prestige nails it[1]
Peer reviewed citations are excellent, and unstable citations like Wikipedia are obviously concerning. (At bare minimum, you'd want to link to the page history instead of the page.) But the willingness to cite arXiv content without hesitation definitely suggests that peer review is only part of what's going on there.
[1]https://twitter.com/adversariel/status/1054836640307392512
There's probably no great answer here, it just feels like authors and editors sometimes use "it was reputably published once" to shrug off disappearing sources.
a) Tell others were to go if they want more context.
b) Give credit for where you got the idea or for doing the legwork of putting it together.
c) Allow auditing of the ultimate origin of a claim and e.g. identify whether one data point is getting repeated like it's two data points. https://en.wikipedia.org/wiki/Information_cascade
Thinking about what I do. There's a difference between me depending on the results of a survey and hearing a novel take on a market or whatever. In the former case, I'm dependent of the survey methodology being sound (and I'm going to trust some sources far more than others). In the latter case it's more "I never thought of things that way" but I'm going to do my own analysis.
Which is sad because most peer-reviewed material are not trustworthy either.
https://bigthink.com/neurobonkers/believe-it-or-not-most-pub...
https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
It's worrying that academia is slowly turning into a rigid "appeal to authority" church rather than a space for truth and inquiry.
It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do.
One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism.
Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter.
Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite.
[1] Personal communication with various scientists, 2018
Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is.
[0] https://academia.stackexchange.com/questions/81261/reviewer-...
So while this definitely does happen (and to different degrees in different fields) there are some checks and balances, too.
I could be naive, but my impression was that there are almost too many degrees of freedom in the peer review process that could allow for this kind of chicanery, and that it would almost be preferable to have criteria for acceptance laid out in advance (i.e., exactly what steps, metrics, and citable articles were necessary) and use those the way an actuary would. In some ways I almost think that practices like preregistration serve as a better first-pass indicator of whether or not research is publishable.
I personally would like more use of pre-registration (there is something about that on the front-page right now. But also I think deanonymzing peer review and publishing the reviews along with the papers could be very interesting. Obvious drawbacks as well of course but I think it has potentially massive upsides.
The papers check results in fields like medicine and psychology which are plagued by the fact that they have to rely on various kind of effects from random sampling of study participants and statistics. I am not surprised that a study with 10 participants leads to wrong results with a significant probability.
But this does not generalize to science in general. Physics and chemistry do not suffer fron this because their experiments have fundamentally different setups. So extrapolating from medicine to "most science is wrong" is just as wrong. You need to approach these fields differently when you want to know how many wrong results are published.
Seeing as how academia literally started out as an accessory to the various Christian denominations for educating priests, I don't see how you really could characterize it as "a space for truth and inquiry". It's only been relatively recently that "appeal to authority" has given way to more secular/scientific ways. Even within the last couple of years, I would wager that academia is no more authoritarian than it has ever been- tenured professors have almost always had a disproportionate amount of influence over their trainees' futures, etc. The head of my father's department when he was getting his PhD in the 70s actually attempted to pull my father's stipend because of an avant garde project that would today be accepted as normal.
I've seen a trend towards the opposite, actually, with graduate students becoming more skeptical of the institution and more proactively addressing its shortcomings through grad student unions and advocacy of open science policies (which neither university administrations nor longstanding tenured professors are fans of in general).
Academia. And even that is error prone due to bias.