How to Read a Paper (2016) [pdf] | Hacker News Reader

How to Read a Paper (2016) [pdf]

blizzard.cs.uwaterloo.ca

372 pointsjdale2740 comments

> The first version of this document was drafted by my students: Hossein Falaki, Earl Oliver, and Sumair Ur Rahman.

The paper is two pages. Why aren't these people included as authors then...?

I was all ready to mock this by saying how most papers get only 1) Read the abstract, 2) Read the conclusion, 3) Look at the graphics, from me.

Turns out that it's basically what the paper says, but then goes into more detail about going into more detail. Worth at least a second pass :)

It's how I got through over 10,000 papers on IT and INFOSEC. When they're well-written, I could look at the abstract, a select bit of details, related work, and conclusion. Would take me about a minute or so a paper once I got good at it. Sometimes less. Some papers were more complex or just not well written where it took significantly more time to evaluate them. All in all, though, the speed reading methods save a ton of time.

I assume you're talking about a time frame of many years.

At least six. Maybe more. Obsessively, too, as it was my grand challenge problem to solve all of it with a holistic solution. Got pretty close conceptually from a number of angles. End result was software or especially hardware options will need to be developed by specialists at significant cost. Now, I still collect the science while looking at marketing or development strategies a bit more.

It typically takes much longer to actually understand the contribution of a paper in engineering fields (EE, for example). Of course, it depends on how much background you have prior to reading the paper.

The background accumulates as you thoroughly read them. You can understand a lot without knowing all the specifics. The good papers will usually explain the challenges, approaches so far, their weaknesses, their solution, and future work that's essentially the solution's weaknesses. One can get pretty far with that. Even formal verification papers were easy to follow on general idea of their methods or results without know the specialist stuff. Same with digital design. I only started getting slowed down, ineffective, or stomped when it was about analog or RF papers. It's why I keep saying one might not be able to cheat those. Not entirely, anyway: my abstract predictions of techniques like A2, the fab material mod, semi-automated synthesis with stochastic methods, or chaining gates together in tamper-resistance show my almost detail-less mental model is still better than nothing. I already had mitigations for two just guessing something might happen with that mental model. ;)

True, but reading the intro and background/prior work sections takes approx. 5-10 minutes. And depending on the length of the paper, these sections may be pretty succinct and therefore not very helpful to a reader unfamiliar with the state of the art.

I agree. You can usually understand the general contributions and how they work at a high level pretty easily. But intuitively and deeply understanding a paper takes much more time from my experience. Again, YMMV.

I also have almost zero experience with analog haha! I'm going to be taking an intermediate course on analog circuits next semester. I'm interested in jumping into the field of analog hardeare security, so it's a necessary first step.

I think you're correct in your intuition.

I recall watching a recent talk by Andrew Ng who mentioned that reading research papers is a key method in improving as deep learning specialist / data scientist, but that one should temper their expectations of knowledge acquisition. One might spend a Sunday trying to go deep into a paper, but at the end of the weekend that person may barely get much out of it. He even admits that sometimes it takes him a couple reads over a course of several sessions to fully grasp a new concept. Which isn't all that surprising given the information asymmetry between the authors and the reader.

If you've ever read a STEM textbook that gives references, you'll often find an entire paper or PhD thesis condensed into a sentence...

This is the first phase of syntopical reading discussed in "How to read a book", which is a great book about reading other (mostly nonfiction) books.

I was about to mention the same book and the same idea out of that book. It's written by Adler and van Doren.

This has some nice hints on how to read the text of a paper, but I think it misses all the important things that you need to be aware when you want to understand a paper. In particular, things like the research paradigm or research programme in which the paper fits, its historical context etc. I wrote a post about this recently: http://tomasp.net/blog/2017/papers-we-scrutinize/

On the writing side of things, I really enjoyed this talk: https://www.youtube.com/watch?v=L_6xoMjFr70 "How to Write Papers So People Can Read Them"

My advisor (Voicu Popescu) wrote a couple of nice materials about "How to Write a CS Paper". The emphasis is mostly on how to do it in the computer graphics field, but perhaps it's applicable to other CS domains too:

In prose form: http://webarchiv.ethz.ch/digitalartweeks/web/uploads/DAWIntr...

In slide form: http://spaf.cerias.purdue.edu/classes/Popescu.pdf

This part was most interesting to me:

"Incidentally, when you write a paper, you can expect most reviewers (and readers) to make only one pass over it."

I understand reviewers are busy, but we depend on peer review to filter out bad or poorly-researched material. I don't think one pass is enough.

Obviously, so does the author.

Depending on the conference or journal, most papers can be rejected in one pass. Heck, the reviewer might abort after the intro if the paper is particularly bad.

The papers that make it past one pass get more scrutiny.

I've gotten reviews where it was obvious the reviewer didn't read the paper. "You didn't address X at all" when I had a section heading under Discussion labeled X in big bold font.

The review process is stochastic and leans towards reject by default.

Indeed, but then this is somewhat to be expected when the peer review process itself doesn't have much in the way of incentives for the reviewers. Journals ask reviewers to voluntarily put their own work on hold for a potentially non-trivial period of time to grok a non-stop fire hose of papers, just so their and the original authors' efforts end up buried under a mountain of other research behind some absurdly expensive paywall later. And if you regularly slip up on deadlines or opt not to play along with these requests for some reason or another, good luck maintaining good relations with that journal in the future for your own papers.

Morals and civic duty aside, it's hard to blame reviewers for skimming or 'phoning it in' under such conditions. With all these subtle politics and screwy incentive structures, I'm unfortunately not surprised at all that science is having a replication crisis.

More practically, one should not expect that a peer-reviewed paper is correct.

For a YC-compatible analogy: think of it as code review. Pure junk gets filtered out, but bugs inevitably remain.

I thought that was going to be Trisha Greenhalgh's How to Read a Paper: https://www.amazon.com/How-Read-Paper-Evidence-Based-Medicin...

Superficially the same idea, but it is for advising medical practitioners on how to apply research to their work. It goes about this by showing you how to find research, critique it, analyse it, use meta-research, and so on.

For somebody not in medicine, it had some transferable advice on how to use research in practice, but was mainly a detailed insight into how evidence-based medicine works. Highly recommended.

If papers were written better, you might not need to read them over 3 times.

There should be a paper called "how to write useful headlines for your paper."

"first pass" "second pass" "third pass" should be replaced with unique, useful, preferably memorable headlines. For instance: "Quick scan" "Deeper but ignore details" and "Challenge every assumption in every statement"

Then you haven't wasted the attention those big bold headlines get.

This is not true. Some of the best papers ever, I have benefited from reading many times, more than three.

Obviously a different set of circumstances, but I would be curious to hear what people think about Cal Newport's Question; Evidence; Conclusion - method of reading. From the grade A students guide. I recently switched to it and found myself understanding reading assignments much better. It seems like this method is more geared for Researchers though.

I'd like to present a fourth option, the "0-pass":

Don't read most papers. Don't feel bad about not reading them, because in general they are terribly written.

Instead, read follow-on work which resynthesizes the ideas in these papers for a popular audience.

You have chosen the path of the laity.

Agreed. Most research papers are a headache to read, even in a field you are familiar with. The full list of reasons is here:

http://www.wikipaper.org/p/Why_So_Difficult

The site above ^^ has not launched yet, but should help alleviate the problem.

The site has launched far enough to attract spammers. It looks like you've got some cleaning and monitoring to do:

http://www.wikipaper.org/p/Special:AllPages

Bookmarked your site, looks interesting for this non-academic.

One of the key questions after the first pass is "is this well-written enough to keep reading?"

Another advice on the topic.

http://muratbuffalo.blogspot.com/2013/07/how-i-read-research...

I sometimes do an exercise with my students where I give them a paper and ask them to tell me in 5 minutes what is the research question and the answer. Usually works well (if the paper is well written), and shows hem that they can get a good grasp of papers without spending hours reading them.

My experience from journal clubs is that strategies like this just encourages parroting whatever is claimed without thinking critically.

For people in the field who are active authors and reviewers in CS/AI related areas:

- How many papers do you read per week?

- How many hours you spent per week?

I first-passed this paper, and realized I needed only a first minimal second pass. Check what to do on the pass on that I on. Very helpful for new readers.

Well this would have been very useful 6 months ago, when I started working on my undergraduate dissertation.

This went way too meta for my procrastinating brain.

I only got to 2nd pass for this paper.

I found it interesting in a meta sort of way that the paper's first pass ctechnique can't be applied to itself (there's no conclusion).

these "how to read articles" make me feel so illiterate -- having completed my formal education 4 years ago. of course, the truth is we do not read closely, and there are always new signals to look into

I see a recursion in this post :p , Before learning to read other papers, could you please explain how to read this paper :D

Reply on news.ycombinator.com