Why Academic Papers Are A Terrible Discussion Forum

Academic papers today are not meant to be discussion forums. To be fair, undergrads and early-stage graduate students often use academic papers as a way to figure out what kind of debates are taking place in their field. But once you get into advanced levels, academic papers are little more than archives of discussions that you had many months and years ago. All the real discussion happens in labs, offices, colloquiums, conferences, by e-mail, and through countless other channels that everyone knows are much more efficient than academic papers. Only those who are relatively isolated from the rest of the academic community, like scientists working far away from major research centers, rely on academic papers to catch up with what everyone else is up to.

Also, sometimes it doesn't matter that the paper gets published a few months or even a couple of years late. Not every academic field progresses at the rate of cutting-edge science. In my philosophy PhD dissertation I criticized works that were 10-16 years old, and even older if you count how long they took to be published. But those works are still considered a hot topic in their field. Hardly surprising in a discipline that has been moving slowly and steadily for thousands of years. The same might be said of many disciplines in the humanities and social sciences, though perhaps to a slightly lesser extent. So whether or not papers can be a suitable discussion forum depends on the field and how people expect their discussion forum to work. Blanket statements that cover everything from philosophy to mathematics to the latest nanotechnology are not helpful.

Some of the other problems you mention, though, are real.

This article seems terribly confused. Of course academic papers are a terrible discussion forum. They're not supposed to be any kind of discussion forum.

Academic papers are supposed to be contributions to science. They take by definition a long time to publish because they need to go through a peer review process, and they are only interest to a narrow group of people because they typically advance the understanding of a very narrowly defined problem deep within their field.

Academics don't just sit in a dusty office communicating with the world through journal articles. We meet in conferences and work in joint projects, have informal discussion forums and correspond through a multitude of different channels. Many academics also have non-academic publications about their research.

Academic papers serve a very specific purpose and they do it well. Just because they don't serve some other purpose that you'd like them to serve doesn't make the process broken.

From "How To Do Artificial Intelligence at the MIT AI Lab" (http://www.cs.indiana.edu/mit.research.how.to/section3.4.htm...):

After the first year or two, you'll have some idea of what subfield you are going to be working in. At this point-or even earlier-it's important to get plugged into the Secret Paper Passing Network. This informal organization is where all the action in AI really is. Trend-setting work eventually turns into published papers-but not until at least a year after the cool people know all about it. Which means that the cool people have a year's head start on working with new ideas.

How do the cool people find out about a new idea? Maybe they hear about it at a conference; but much more likely, they got it through the Secret Paper Passing Network. Here's how it works. Jo Cool gets a good idea. She throws together a half-assed implementation and it sort of works, so she writes a draft paper about it. She wants to know whether the idea is any good, so she sends copies to ten friends and asks them for comments on it. They think it's cool, so as well as telling Jo what's wrong with it, they lend copies to their friends to Xerox. Their friends lend copies to their friends, and so on. Jo revises it a bunch a few months later and sends it to AAAI. Six months later, it first appears in print in a cut-down five-page version (all that the AAAI proceedings allow). Jo eventually gets around to cleaning up the program and writes a longer revised version (based on the feedback on the AAAI version) and sends it to the AI Journal. AIJ has almost two years turn-around time, what with reviews and revisions and publication delay, so Jo's idea finally appears in a journal form three years after she had it-and almost that long after the cool people first found out about it. So cool people hardly ever learn about their subfield from published journal articles; those come out too late.

You, too, can be a cool people. Here are some heuristics for getting connected: ...

TL;DR Academic papers are for presenting final results and assigning credit, not discussion.

With respect to item #1 (huge time lag), it's worth mentioning that in some fields—I'm familiar with psychology, neuroscience, and biomedical engineering, but this likely applies to others as well—many if not most labs are in some form of contact during the course of the year, whether it takes the form of planned or serendipitous meetings at conferences, email conversations, second-hand talks through mutual collaborators and similarly-minded faculty, or other channels. In these fields, it's pretty unusual to be working in a total vacuum, where your only method of communication is through academic publications.

I think the author is implicitly talking about academic philosophy papers.

Regarding point 8, the "bottom line" rule (http://lesswrong.com/lw/js/the_bottom_line/) is about process, not actual placement of the bottom line. It is about asking how the conclusion and evidence came about: is the conclusion derived from the evidence, or was the conclusion stated, and the evidence filled in afterwards to support it?

Sure, I will state my conclusions in the abstract of my (computer science, mostly systems) papers. But I arrived at those conclusions after designing and experimenting with the artifact I'm presenting in the paper. That my conclusion comes first in the paper is not what matters.

I assume the author does not mean to apply point 8 to engineering and scientific papers. But I want to point this out since he didn't say so.

Academic papers fail as a Reddit-style discussion forum because they were never meant to be a Reddit-style discussion forum.

If you need to discuss things with people, join a mailing list or go to a conference or get to know them. The paper is not supposed to be the transitional step where you are bouncing ideas off people.

I agree that academic journals should be open access. I don't agree that it is unfair for subject matter experts not to publish every crank who comes down the pike in every journal.

If you want to talk about enforcing conformity as a bad thing, a discussion forum (particularly one with upvotes and downvotes) is the last thing you should use as a model.

I guess you are more directly engaged with Less Wrong and Singularity Institute and other online fora which lack any academic credibility, but sort of pretend to have some.

Who claimed that the primary purpose of peer-reviewed academic papers was to serve as an all-inclusive, real-time, 'democratic' discussion forum?

Next up: "4chan flame comments are a terrible way to incrementally advance the state-of-the-art in a narrow academic field, presenting theorems, experimental data, and citations to closely-related work."

Say what you will about academic papers, but I needed to know how to solve a tridiagonal matrix efficiently on a distributed computer, and damned if I didn't find some that told me how.

As a survey of the field, maybe papers in journals aren't so great, but as a way to find heavyweight answers to heavyweight questions? You bet your ass they're good.

(Of course my University pays through the nose so that I can do that, but that has no bearing on the format).

There is a reason why conferences are so hugely important. Often, what will be published as an article is presented beforehand on conferences, among knowledgable and interested peers. And while some conferences are invite-only or have to be paid for, most universities have a lot of conferences either completely open to the public or at least open to students. Often, just asking someone of one is allowed to just audit the talks without going to all he social events aimed at invited people will be enough in order to be "let it".

Overall, what bothers me most about this blog post is this:

>> Papers are, intentionally, written for an audience of specialists rather than a general interest group, which reduces both the tendency and ability of non-specialists to read them when asked

This repeats the same bias that everyone can say something about everything. Nobody here would complain that mailing list discussions about bugs, features or programming questions are hard to understand for non-programmers. However, a scientific discussion in a paper is supposed to cater to everyone. In reality, especially within the confined spaces of an article, one has to assume some baseline of knowledge in the reader, because one can't afford to explain all the basics one implicitly builds upon. To give an example: I do literary studies, and often people assume that they know just as much without having properly dealt with it, because hey, everyone can read a book, right? right? While that part might be true, every scientific discipline over time builds up a plethora of instruments for analysis of their respective topics, an understanding of the matter's and the subject's history, its pitfalls, shortcomings and strengths.

Also, I do not understand his paragraph regarding the "bottom line", since I know noone who writes the abstract first. The abstract is the very last bit to be written, after the paper's structure, arguments, theses, supporting evidence and rebuttals are decided, fixed and usually written. What both the cited Paul Graham essay and his text fail to see is the process of writing a paper. It is true that the actual writing process usually doesn't start with a thesis statement and a chain of arguments. But structuring a paper is just as much part of writing as finding words for what one wants to express.

The author hints at being an undergrad in maths, having trouble with the understanding of state of the art research. A professor I know once said this about degrees: undergrad & grad work are a necessity, they teach how to start a car, how to accelerate, break and also drive on basic roads (research methods/skills). A PhD then allows (and teaches) you to drive on the autobahn (research), where you can start to close the gap somewhat to all the people already on it going faster than you do. The actual entry into novel research doesn't happen until after the PhD. That's where journals and conferences, workshops, groups, funding etc. become interesting and important. First then are you within the actual target group for academic papers, since you are seen as an equal, equal also in terms of discussing on eye level.

Nobody expects complex technology etc. to be understandable by everyone, but one could say: "Everyone thinks they are a writer, much like many people stand in front of contemporary art saying they could produce the same thing." What they fail to understand is the complexity involved in the writing process. What sometimes looks easy or even "trivial" in a finished paper often has a highly complex, dense and far-reaching background.

[edit] Mhm...I apologise for the length of this comment, it grew larger than I originally thought it would... [/edit]

The real problem is that 99% of research publications are, to put it bluntly, worthless. This is the root problem of the entire system which leads to many of the problems the author here is listing.

These 99% are often churned out because they help sustain someone's carrier or lab. To be published, crap research thus needs to convince people that it is smart and important. This is usually done by being written poorly.

The remaining 1% will progress human knowledge and help other scientists to continue down the right path. If they are important enough for wider dissemination, well, that's what science journalists are for.

> 8. [...] one starts with the conclusion in the abstract, and then builds up an argument for it in the paper itself.

Yeah but you write the abstract last (intro too). Having a conclusion at the beginning is just UI.

This post might be more useful if it offered more specific alternatives, rather than mostly criticizing the academic process. Imagine the same ideas reworded as suggestions for improving the process. Then the posts here could discuss the new ideas (useful!), rather than debating the criticism (tedious!).

Perhaps peer review should be done in public, as a page-rank style rating according to the authority of each reviewer in that domain. Would it be completely open, single blind? double blind? Good questions. But it might reduce delays and decrease the effect of politics on the process.

I think the central urge in there is to get scientific discussion more out in the open. The single biggest issue by far, is not the pace of the publication process or the level of writing. It is access.

I would argue that access would actually solve a number of his other complaints (poor moderation, percieved time lag, people don't wanna read 'em, academia selecting for conformity). If we raised the quality of content on the open internet, the quality of discourse would go up.

Could this be a niche for a software solution enabling scientists to comment/share/contribute on each other's papers faster and more efficiently? Perhaps in the end speeding up research times and in the grand scheme of things furthering the scientific process?

HN outperforms on all those points

This article has some good points and some bad ones too.

e.g. Academic papers violate the "bottom line" rule.

If you were making stuff up as you write it down then it would be true that starting with a conclusion might bias your arguments towards justifying that conclusion. This might describe how a (bad) journalist writes, but not how scientific papers are usually written.

The majority of the work for a scientific publication usually comes well before writing the first word. Theorists will discard many theories before they find one promising enough to do extensive work (primarily aimed at proving that theory wrong) before they even consider publication. Experimenters must work with theories that have been through the above, conduct a complex (and usually expensive) experiment, and then analyze the results extensively before they start writing. The writing process can trigger some consistency checks and digging through references may reveal new spins for or weaknesses in the theory or experiment. However, if the entire point of the paper was wildly misguided to begin with, it's unlikely writing the paper in a different order (so that the authors aren't beguiled by their conclusion) will be what reveals it.

Now, the big benefit to writing scientific papers with the conclusion presented right up front is that you don't have to read entire papers to figure out what they're talking about. Scientists generally don't read papers to be led down a grand story with a surprise payoff at the end. The first thing they want to know is what this paper has to say so they can decide if it's worth reading. Science is big. Really big. Go to arxiv.org (a preprint repository for both articles that will be published and some that won't). Pick one specific sub-field and look at how many papers were submitted yesterday. The reason few people read material from outside their bailiwicks is because they barely have enough hours in the day to keep up with their own field and do their own work. If scientific papers followed the "bottom line" rule things would get much, much worse.

Personally, my biggest complaint about journals is how expensive they are. The only reason university researchers have "free" access to journals is because the universities pay (frequently exorbitant) subscription fees to the journals. It's better than having every researcher pay $30 every time they want to look at a paper, but it's still far from cheap. The really weird thing is that it costs several thousand dollars to publish a paper in a decent scientific journal. Yes, you do ground-breaking work, write a beautiful paper about it, and then you have to pay the journal to publish it if they accept it, and color figures are almost always extra! Where does all the money go? Journals have remarkably few expenses. In terms of producing content the only people they really have to pay is their editors. The authors pay and the referees work for free. Publishing the content does require some infrastructure, like servers and code. The print version requires type-setting, production, distribution, etc... However, I find it very difficult to believe that journals need to charge what they do to either creators of the content or the consumers of it. I'd love to see journals open up their books and reveal where the money is going.

Upvoted!

Wish I could do that for academic papers.

Academic papers also make terrible pornography. That's not what they're designed for either.

Well, there's at least one way in which academic papers are a great discussion forum: People think a lot before posting and do thorough research on their arguments - and those of their opponents.