No Paper Is That Good
econlib.org
econlib.org
This is one of worst ideas I've ever read. Wouldn't you just be cheating yourself?
How is tying your own learning to someone else's ability to find the best papers in any way a smart thing to do? It would be much better to doubt the person who gave you the papers (perhaps he miscalculated which two papers were the best), than to dismiss the entire field.
You stumble on a bunch of psychology papers in which psychologists noticed that whenever 6+ kids get together, there are either three mutual friends, or three mutual non-friends.
The psychologists think this has something to do with child psychology and have written 10,000 pages on it.
You immediately recognize it has nothing to do with psychology and is just basic Ramsey Theory [1]. Should you have to review thousands of pages before being allowed to chip in?
[1] https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(...
A less hypothetical example: You're Bertrand Russel and you discover Russel's paradox, a one-liner which negates thousands of pages of Frege's not-yet-published logic textbook. Do you have to wait for him to publish it, and then you read it, before raising your voice?
But there's a crucial difference between that and the way original rule is phrased.
The fields one has to dismiss are characterized by things one knows is false by physics or combinatorics or other common sense things. Theories of telepathy, the flat earth, creation science or whatever.
The "you can't show me one good paper on the subject" is an appealing-seeming pronouncement but it's really dumb way to do it. Just say, "that doesn't make sense and you'd need huge evidence to prove it". Paper quality isn't really the question.
https://www.sheldrake.org/research/animal-powers/a-dog-that-...
In one of his experiments monitored by another scientist, when the dog didn't respond in a way that met Sheldrake's criteria, Sheldrake changed the criteria in mid-experiment.
That one incident alone should be enough to dismiss any results coming from Sheldrake. He's not only dishonest but blithely unaware that he's being blatantly so.
If you mean that 10000 pages in total in 10-page papers, you have read the literature, and become the hero child psychology needs by writing a paper refuting this.
On the other hand, if you did not read the literature wider, how do you know that you are first to do so, or that the misconception is even shared by other than the authors of those papers. Of course, the implicit assumption in all this is that psychologists are stupid.
R(3,3) doesn't seem universal enough here to be touted as "isomorphism"; it is not giving any particular insight except the most trivial (that child cliques sometimes can be represented as monochromatic sets of a complete graph). Am certain one can do a number of other graph-theoretic or algebraic relationships here with zero insight or predictive force.
So yes, if kids' friendships form a complete undirected graph, then sure, your result is not at all surprising. But maybe they don't.
On the other hand....Ramsey’s theorem applies to compelete (undirected) graphs. Friendship is not necessarily symmetric: Alice could consider Bob a friend, but not vice versa. It could also be context-dependent, especially with kids: Alice and Charlie get along, but not when Bob is around.
The thousands of pages that you don’t want to read probably address some of these “details” that your spherical cow model totally ignores. I think it’s totally fair for someone to say “That’s interesting, but have you read X,Y, and Z? They show that your model doesn’t apply because....”
My bigger point was that the literature on any given topic is chock full of discussions like this. Is friendship directional? Is it binary, or does it make more sense to consider weights on these edges (casual acquaintance/best friend)? Can an outside observer infer friendship, or do you need to rely on self-reports? If so, how consistent are they? And so on.
The more patient researchers are happy to walk you through these sorts of considerations, especially when you're introducing them to a new tool or something. However, if you catch someone who's busy—or grumpy—you might get blown off with "That won't work. Haven't you read the literature?" Personally, I think this happens too often, but you can sorta see how people might get sick of regurgitating the same arguments over and over.
This is not to say that "read it and come back" can't be used as a moat, or that ideas "from the literature" shouldn't be revisited and questioned, but I think insisting that people do a bit of reading is not usually meant in bad faith.
You always need to read all relevant literature to be able to constructively participate in a debate.
Non-adherents can't attack an idea because someone simply says "well have you read X paper? No? Well, then you need to read more.".
And then when you can back and say "but this paper is wrong too", the process just repeats.
However, if you show you haven't engaged with the relevant literature and don't address the arguments in sufficient detail, then your paper is likely not going to pass peer review, and rightly so.
Although, the original paper of an idea, I often find, has the most human-readable explanation. Pearl's 1986 paper of Bayesian Networks, Gauss' 1800-whatever paper on the normal distribution, the original bitcoin/blockchain paper by that Japanese-sounding name, etc. They're still trying to sell the idea so they really need to talk to the idiots.
In my opinion, a real intellectual is interested in the topic and assorted problems and will critically read whatever fuels their interest. Even papers that are not good can be interesting and worthwhile reading. Wise men know the principle "garbage in, garbage out", though, and that their time during life is limited, so they choose their readings carefully. However, that doesn't mean that they shouldn't include fairly bad literature, too. For example, it's necessary to read seminal papers that are part of a canon, no matter how stupid they seem to be.
When I started studying philosophy a long time ago, some tutor whose name I've forgotten gave us two pieces of advice that would have been very valuable to me if I had taken them seriously earlier:
1. Never dismiss a book or paper quickly because you believe you've found a mistake. Read them until the end and take the authors seriously!
2. Aim for writing 1 DIN A4 page a day, no matter what you write (could be personal, could be professional, doesn't matter). You will write much less but as a goal about 1 page a day is good.
For example he says "full of mistakes", not one mistake.
There is nothing wrong with choosing your literature carefully, but that's far from the original suggestion. You can and should use your bullshit detector.
Because of this formulation, the standards are set rather high. Bryan Caplan's point is that if the best two papers out of a field of hundreds are not above these standards, then there is little value to be gained by slogging through the hundreds.
>Bryan Caplan's point is that if the best two papers out of a field of hundreds are not above these standards, then there is little value to be gained by slogging through the hundreds.
And why would anyone need criteria for that? What kind of person would need to justify to themselves or others that they prefer to study topology instead of, say, financial mathematics?
Caplan apparently needs some criterion to be able to be dismissive about someone else's work or discipline rather than pursuing his own scientific goals and interests. Yes, I find that kind of anti-intellectual, or at least small-minded.
> And why would anyone need criteria for that? What kind of person would need to justify to themselves or others that they prefer to study topology instead of, say, financial mathematics?
We need criteria to assess the value to papers, because generally the whole assumed point of writing a paper is to add value.
> Caplan apparently needs some criterion to be able to be dismissive about someone else's work or discipline rather than pursuing his own scientific goals and interests. Yes, I find that kind of anti-intellectual, or at least small-minded.
Ad hominem much. Also, Caplan doesn't say that, and you are talking about the original proposal. You don't seem to have read the article further than the title and the premise, though interestingly you seem able to judge the author itself and convict him with bad thinking. This very much contradicts your own previous piece of advice
> 1. Never dismiss a book or paper quickly because you believe you've found a mistake. Read them until the end and take the authors seriously!
which, to me, seem rather like virtue signaling than a real advice; all the more when you don't apply it thoroughly in reality.
I simply objected to the idea on the linked web page that you can ask someone to give you the two best articles in a field, then assess these, and use your resulting opinion to somehow evaluate the whole field as an outsider.
That idea is silly and small-minded. If that feels like an ad hominem attack to you, so be it.
It's not - it's a realization that life is short and much research is just writing.
If they are supposed to be an expert in the field, that seems like a fair test.
[1] A Mathematical Theory of Communication - http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
I was motivated however to install an archway for my house from the exact shape in Figure 7 (binary entropy function)
The paper contains citations. But they are hidden in the footnotes - see page 1 or page 5 for example.
Erdos has a few papers that are "that good" as long as you know all the theory leading to the new insight, but none (that I read) are as clear and freestanding as Shannon's AMToC
>Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies.
https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...
Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever.
I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place.
I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.
A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened.
Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless.
In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this
This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.
I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.
I think that psych should be commended for attacking this problem. As a field they are funding replications, which basically never happens in any other field. We don't even know what the "replication crisis" would be in empirical CS. Given discussions with ML people I'd wager that the "replication crisis" in ML is at least as dramatic.
Science is messy. But this is why we should defer to experts who have vision over an entire field rather than focusing on individual papers as conclusive data points.
Psychology has a history of being sort of self-analytical (which makes sense if you're a field studying human behavior). Meta-analysis, for example, was largely incubated in psychology before it started expanding into other disciplines.
The criticism I'm aware of is really being leveled at the idea that this is somehow unique to psychology. It's like killing the messenger, or someone who is trying to fix the problem.
It is understandable though. Once the paper is published, then it is no-longer novel, and there is obviously lack of reward and motivation to write the same idea again -- just to write it better. ... unless you are actually writing a review paper, but then, it appears there is little value unless the review is "complete". A good illustration of the idea need be able to high-light the key idea while avoid having minor ideas obscure the key. Therefore, a "complete" review paper rarely provide a good read.
Compared to papers, a good text-book often is a much better read than all the original papers. It is not really because of the size, rather it is because a good text-book is written from the reader's point of view and focuses on conveying the idea itself (vs. selling the idea). And a good text-book takes many rounds to develop.
There is no reason papers cannot be developed in a similar way as text books: once a ground-breaking paper is published, it shall be constantly updated, each new edition reflects what the author has newly learned and incorporating new development including the entire community...
Alas, that is not the culture, and there is no motivation to do such.
I think that courses remain the best way of distilling and communicating knowledge. The only problem is that they're not always available to a lay audience, since there's little incentive to make them available.
The other alternative for the lay audience is science/economics journalism (Economist, Scientific American, Discover, etc). This works to some extent but mostly just scratches the surface, since even regular journalism is struggling to be profitable these days. With deep technical topics there are too few readers and too few qualified, willing writers.
Fifth, most researchers’ priors are heavily influenced by some extremely suspicious factors.
Academics in the humanities identify themselves "I am an X." X can be post structuralist, rational materialist, classical liberal or some other broad, hairy, intellectual identity. This is bad news for objectivity. Everyone has a dog in the fight.
This other stuff seems to be about "what is useful to us as humans in a group matrix (culture/community". That seems like it should be studied - but I don't think we all have the same objectives. Also - some people keep trying to make it "true" vs. "useful".
What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"?
I don't know - it just seems (to me) that we spend too much time on the truth as opposed to (subjectively) communally useful. i.e. What should we decide to believe in as a community to hang together (because otherwise, we shall surely hang separately)?
The problem with that is whoever takes it upon themselves to discard truth and consider 'what will have the best effect on people/society if they were to believe it' —is that you have to manipulate other people into believing it's true on the hope that your assessment of how it's going to affect people is accurate. It's almost certainly not.
It's interesting considering the human brain's criteria for accepting beliefs. It works pragmatically, not with strict adherence to truth—but, from what I can tell (and I've seen some research supporting this), the amount it's willing to deviate from (what it estimates to be) truth is proportional to the immediate demands of present circumstances. If you are in a crisis, it is willing to compromise and adopt a belief that conflicts with evidence, as long as it solves some immediate problem you're facing. But when that happens, you are accumulating something like technical debt. You become sort of 'out of harmony' with your circumstances, and problems will arise given enough time.
Why would it evolve to act in that way? It seems clear the answer is: because it's generally most beneficial to survival to adopt beliefs which best fit the observed data. It's a fair assumption that society holding true beliefs is also going to be most effective for survival.
So when someone manipulates someone else's beliefs because they think it will be beneficial, they are forcing that technical debt on them, and that's a decision only they should make for themselves. The same is true but amplified when considering spreading these ideas to wider audiences.
That said, much of contemporary social science and the humanities are rife with the opposite sort of thinking. This is heavily fueled by the fact that much of it doesn't consider 'correspondent' truth to be a real thing, so in their framework it is meaningless to say they are compromising the truth. Which truth? You can see this in Pragmatism's approach of literally redefining 'truth' to refer to something like 'the most effective belief to adopt'; and even more pervasive are all the schools of thought influenced by a kind of radical relativism which denies the possibility of truth. (It also clarifies a lot of philosophy once you realize the philosophers are actually thinking about the best mind-programs to put into people, rather than seeking truth as a typical person would recognize it. If only the sneaky bastards would say so to their readers' faces. But of course it would not be pragmatic to do so.)
It's a bad idea, and I sincerely hope the fields presently employing it fall into ruin, or are otherwise convinced to abandon it.
> It's a fair assumption that society holding true beliefs is also going to be most effective for survival.
This is the part I disagree with. People are social. And their environment is other people. Imagine a nation or tribe populated by Aztecs, or Mormons (No hate - just an example). A person's ability to survive in a territory populated by Aztecs or Mormons is going to be significantly determined by their ability to conform to group beliefs re: normative behavior. The easiest way to do that (thanks mirror neurons!) is to believe it yourself...
I agree, but I see that as a separate issue from what I was attempting to address. The specific thing I had in mind was about authority figures in a position to promulgate some new idea to society: is it wise for them to spread ideas known to be false but estimated to have positive effects?
Man may not be the measure of *all" things, but he (she) is the measure of some things and that's ok. I think a lot of arguments (politics and political ideologies in particular) would get a lot nicer if we just accepted that we aren't arguing about absolute truth.
If you’re only going to read one microeconomics textbook read Varian’s Intermediate Microeconomics if you can do calculus. If you can’t Cowen and Tabarrok’s Principles is pretty good and has a load of wonderful accompanying videos.
Comparative Advantage
https://www.mruniversity.com/courses/development-economics/c...
And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows.
https://leaders.economicblogs.org/noah-smith/2017/smith-lite...
I was only half-serious in my original comment. What mud moat exists today is maintained less by economists and more by politicians and businesspeople, who find certain "classical" economic conclusions convenient. "Free markets are good - just read the economics literature!"
But to be qualified to read these papers and appreciate why they are points of quality within a sea of crap? That's hard.
These papers are not exhaustive summaries of a field. But a reader comes away understanding the type of problems a field is devoted to solving and many of the existing ideas. And I believe that each is a paragon of their field.
In a theoretical paper, it’s possible to make statements that stand on their own merits. In empirical science, a single paper is never really enough to support an entire field. Empirical sciences generally rely on the development of scientific consensus.
I try to do the former but the work seems so boring that I am hardly motivated to do this more.
For physics: "Okay, this paper is from so and so, yup, same facility and equipment as the previous one. Theory section, same basic overview, skip to the end for what makes this experiment worthwhile? Anything interesting in their analysis, and how good are their statistics? Did they show the usual set of diagnostics plots for the type of analysis they did, and if not, what are they trying to downplay?"
I've heard arguments that one should leave the abstract until the end, since it sometimes pushes a story harder than the data necessarily warrants, and so if you come to your own conclusion about what's going on in the paper, then check to make sure the authors claim what you feel is supported, you're less likely to be misled. Probably this makes sense for reviewing a paper, but for most things, I'd rather find out quickly if the paper is even relevant to what I'm working on.
You have to read the abstract. It's the intro and conclusion you can skip.
Read the title and abstract. Then read the statements of the main theorems and corollaries (skip the lemmas and interstitial commentary at first). At each step, evaluate whether you want to continue or abandon the paper for whatever reason (not interesting, not relevant, whatever).
Finally, if you made it this far, read the whole paper word for word with a pen and paper (or chalk + chalkboard) handy.
For longer papers or books/theses (say, 20+ pages), do this whole process on each individual section/part/chapter that you want to consume.
It also helps to have a reason to read the paper besides curiosity. I find things that are relevant to current work to be easier to get through than more peripheral sorts of things. Having a colleague to discuss things with also helps.
Edit: Although it may go without saying, all else being equal, a better paper is easier to read than a lesser one. But, really, content is king.
my phd advisor started by looking only at all the displayed formulas. If there was nothing interesting or new, he would not bother reading any sentence of the paper.
That is incorrect. The two-paper rule is Noah Smith's, and Bryan Caplan is disagreeing with it.
(It's scarcely possible to read any of Caplan's post without realising that; I conclude that many commenters here have not bothered to read the OP before commenting.)
From that perspective, the suggestion seems fine. You should be able to dig out two examples that show your field isn't nonsense. I don't think it was meant to be a high bar:
> There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.
If the field is not totally vague, then yes. We can still consider certain papers in physics, or chemistry, or medicine, computer science etc. as exemplary decades, or even centuries, later, even when they deal with empirical work.
Soft sciences need not apply.
(What did I learn? I don't remember precisely - it's been 15 years - but I think it was that something on the electromagnetic side was caused by time dilation.)
Why not? Historians still revere Braudel and The Mediterranean was published nearly 70 years ago. Are you certain that there are no such exemplary works outside of the fields you like?
Nonsense, knee-jerk answer.
First, we don't need all of them to "hold up well to scrutiny" but just 2 (as per TFA challenge). And we have way more than just 2 -- hundreds of great papers.
Second, the papers "that were trying to optimize for bottlenecks in hardware that no longer exist or have shifted to different places" can still be perfectly valid as per the challenge we have, which was:
1) that they were not "full of mistakes and bad reasoning", 2) that they did not "contain little or no original work" (and where thus just references and meta-papers)
The question wasn't if we have "plenty of" papers that are bad, or plenty of papers that were very good but have been super-ceded by changes in technology.
Just that we have at least 2 (and I argue we have way more than two) seminal papers that have original work, and are not full of mistakes and bad reasoning.
People forget the original name for what we call economics was "political economy". That alone should tell you all you need to know about the dangers of treating that field like a science. If you ever want to know why expert economists can't seem to agree on things that happened 50-100 years ago or make accurate predictions for the future, the original name is very telling. Wouldn't it be ridiculous if we were still debating the validity of f=ma or e=mc^2? Wouldn't it be crazy for someone to claim general relativity is just flat out wrong, even though GPS systems would not work properly without humans accepting it?
Why is nothing remotely approaching reasonable standards used in the social sciences before acceptance?
Simple.
Easy to read.
Easy to replicate (a poc in python is easy)
It is a recent key innovation (the blockchain tech).
Does all of this hold for my papers, too? Of course. The most I can claim is that I am hyper-aware of my own epistemic frailty, and have a litany of self-imposed safeguards. But I totally understand why my critics would look at my best papers and say, “Meh, doesn’t really prove anything.”