How to seriously read a scientific paper
sciencemag.org
sciencemag.org
To add more - not reading any text in the paper makes you laser focussed on trying to figure out what the data means without the authors trying to sugar coat anything with their perspective/agenda.
It's a question of who you prioritize while writing a paper. Experts or the interested layman.
If you prioritize people who are already experts in the topic, you write little to no introduction, you get to the results quickly.
If you prioritize the "curious, interested layman" (or university students in their first years), a short introduction to a topic with references will provide enough information to understand the basics and the reader can continue reading the rest of the paper with enough context to understand why the topic is relevant.
Laymen, for example, might need some figures to help understand a topic that experts internalized years ago.
I prefer having occasional well-written review papers, meant for getting non-experts up to speed. Then the domain experts - who are often not experts at writing for laymen - can refer people to those review papers, possibly also with a history delta for what's new since the review.
"For a comprehensive review, see" https://scholar.google.com/scholar?q=%22for+a+comprehensive+...
That's what surveys are for.
My field of cheminformatics, while CS-adjacent, inherits more strongly from the chemistry traditions. The occasional outsider papers from a CS department generally cause my eyes to glaze over if they follow that history-retelling CS tradition.
It's much better than having to look through all the relevant papers in that topic in the last decades in order because they all assume knowledge of the past ones.
Of course, I know there are survey papers, and they can be very useful, but they're typically going to be too general and not specifically oriented to what you need to understand that specific paper you're interested in. Plus, you inherit the biases of whomever compiled the survey, which in my experience are often significant.
For example, in paper I reviewed, the CS authors misread one a paper completely. They wrote something like "method X has been used in cheminformatics before [cite] but the CS literature has improved on that with method Y." But the paper they cited actually used method Y.
Now, the paper itself didn't need that level of detail about the history. That error, and others like it, bugged me because they came across as dilettantes, writing with more assurance than they actually had, and because their history was biased towards the CS methods they knew, which made it feel like they snubbed the cheminformatics methods and treated the previous work in this field as second-class material.
I can't help but think that the process you describe, where someone new to topic must write a history section just to publish, results in a lot of half-baked history, as new people just don't have the experience to really give a good treatment of the history. Instead, they'll see that 15 other papers covered points A-F so they follow the tradition that they need to cover points A-F but with a different slant.
I'm not saying my field is immune to that! There's a well-known observation along the lines that "similar structures tend to have similar properties." Many people will cite a 1990 as the source of that quote. Except that that book doesn't contain that quote. Most people instead know about it second- or third-hand, which has resulted in the common but incorrect practice of making that citation. It's a litmus test I use to tell if the authors really know their history.
Q: If you write multiple papers on a new topic, do you still write histories for each one? Or can you refer to your previous publications for the history?
And the answer is that in general, we do write (short) histories for each paper, unless maybe in short conference papers (limited to 4 pages or so) while it's OK not to write any.
It's not really about history so much as context. Usually, you want to set up a paper with "This is the state of X as it exists right now, but there exists this problem Y. We solve Y by starting from X and making Z advancement."
I think part of the issue is CS is exceptionally young (less than 100 years total, really), and the other part is how rapidly it's advanced and diversified in that time (in terms of individual disciplines even within subfields). Without the context, I can't readily jump to a paper from an adjacent-but-not-directly-relevant subfield without needing to look up a bunch of other stuff. And it's not as straightforward to know where to look to find the relevant information. A paper 10 years old might be state-of-the-art or it might be completely outdated, and it's hard to know which if you're not immersed in that discipline. Having the context in the paper itself is a big help to ameliorating this.
Is it possible they were secretly reporting something else too?
The process was utterly a time waste except for when the paper in question was a survey paper.
now you have sixty+ complex papers, at least a third of which are not actually very important, useful or thorough.. and where are those original, carefully chosen ten you started with ?
side note - the "focus on the figures" reading advice does not scale, since most search is first and foremost with text. Which of the now-eighty papers (and growing, the field is hot) are the ones you cared about and understand.. ? "piled higher and deeper - PhD" indeed!
I've been damned lucky. The worst paper, the one were I had to read most of the citations and quite a few of their citations before I got it was http://conferences.sigcomm.org/sigcomm/1997/papers/p011.ps and yes it took over a month before all the pieces settled into the right corners of my mind. Perhaps more accurately, it took months for my mind to create the right corners for the concepts to settle into.
But that was fine - I knew before I started it was the seminal paper on the subject, and so it would be worth whatever effort it took. The idea of wasting that inordinate amount of time going down that path with one dud after another makes me shudder in horror.
Also, I've noticed that a very large range of papers with faked image data in figures has gone unnoticed by most readers. People look at the figures hoping to see what they want to see and aren't critical enough about the process used to generate them (when I wrote my phd thesis, all figures were programmatically generated by version-controlled code on well-managed data.
The methodology tells you exactly what the people who wrote the paper did. It's where you can see whether they used a sample size of 10 vs 1000, what methods they used to sample their data, the accuracy levels they used in analyzing their data, it's also how a study may be replicated.
Without reading the methods, you can't be sure about any other thing in the paper.
That's the problem i see with a lot of news reporting on science papers. They use the gp's method of reading. Abstract, images and maybe results if they're really trying.
That's how you end up with a lot of sensationalist, contradictory science headlines.
The sciencemag article is nice read after this paper though, since there's a couple of small trips&tricks that is not in that paper (such as the emotional side of reading a paper, how to go on when you actually don't understand something/jargons, when to give up/look for better paper, etc)
Different ends but similar means (multiple passes)
Heard another one about airline safety guidelines written in blood.
I used to do this but I've tried to move to exclusively digital. I always, after some period of time, end up throwing out the papers I've printed. Plus digital allows for easier access across devices, quick searches, etc. Still though, I enjoy reading printed papers more.
Curious if anyone else has a similar experience with print vs digital.
One of the most obvious benefits to paper that comes to mind is the ability to lay multiple non-sequential pages (from the same paper and from other papers) side-by-side. That spatial aspect isn't replicable afaict and I definitely feel like it limits me in many ways.
My current thoughts are to move more to Zotero because it also has some annotation functionality, but primarily use it as an extraction point to then store these annotations/notes in plain text or markdown. Which then I'll manage with git or something.
I've personally realized it's more important to me to have my notes about the papers easily accessible (hence plain text in git) than to have PDF annotations stored in an app alongside the PDF. I've also realized this has been a little bit of a shift in how I read, making me more focused on extracting information. Not sure how I feel about this yet.
It takes extra time, and might feel wasteful, but I usually do it some days/week after I read the paper, for a spaced repetition purpose too.
Instead of highlighting I take notes. I write down a list of key points, and I write down a list of topics I'm not as familiar I'd like to be with, including breaking expectations.
I go further and I google around investigating this list learning more on the topics beyond what the paper has to say.
This style may or may not be ideal for you, but it works well for me. Highlighting has always, for me, been not very helpful when compared to taking notes and getting a more solid understanding by combining sources of information on the topic when possible.
Not just reading papers, but Wikipedia as well: When exploring a new domain of knowledge, you may not know much or any of the vocabulary being used. This turns the average person off. I see this as the most common hurdle faced by those self studying a new topic.
The trick is to slow down, look at the different vocabulary you don't recognize, and then look them up. What sometimes happens is to learn this new vocabulary you'll be met with a wall of even more unfamiliar vocabulary. This turns into a dependency chain of unfamiliar topics. Looking up those words / dependencies recursively, eventually you'll get to familiar knowledge. You can start building up from that place of familiarity and before you know it you've picked up not only the basics of a new domain of knowledge but the more advanced topics build up to what you were interested in.
Learning the prerequisites usually takes hours, but sometimes this process can take days. Taking days to learn a dependency chain to learn an advanced topic seems like a long time, but in comparison to taking a class on the topic or reading a text book, this style of learning is highly accelerated. Sometimes there is no alternative learning style if you need to parse a paper on a cutting edge unfamiliar topic not yet covered in text books, and imo if you pace yourself and value learning over any other goal, it can be quite a relaxing and enjoyable process. Reading papers is a lot of fun.
When I'm learning new topics, I take notes. My goal is to maintain what I learn, not have it go in one ear and out the other, so why not slow down, relax, and take notes? Better yet if I come back to the same topic months later I can look over my notes. If I forgot something about the topic, I know my study style could be improved. This meta-learning feedback loop can be super helpful if you value and enjoy learning new topics (which you do or you wouldn't be on YC).
I think there should be scientists and science writers. These writers should be employed by research institutions. Their job should be to understand the concepts and explain them in simple words.
I feel this would give the science world a lot more accessibility.
My problems with science papers have always been that the statistics have been written incomprehensibly (no layman knows what a p-number is) and that they repeat themselves so much that half the paper is just nothing
(Hopefully you can take that as the satire it is...)
https://www.elsevier.com/connect/infographic-how-to-read-a-s...
https://web.stanford.edu/class/ee384m/Handouts/HowtoReadPape...
https://www.owlnet.rice.edu/~cainproj/courses/HowToReadSciAr...
https://towardsdatascience.com/how-to-read-scientific-papers...
https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
1. Read summary/abstract.
2. Then skim intro.
3. Then read discussion/conclusion.
4. Then methodology.
5. Then read front to back.
6. Then read front to back again: but now we're being extra skeptical: we're questioning everything, looking for common bad practices, checking doubtful references to see whether they're true (the number of references that I follow where I don't agree with the author that the reference says what they claim is too damn high).
7. on loop: read from front, hit and study topic/part you don't understand, focus on and study that part, upon understanding continue read through to see whether understanding elucidates any other parts you didn't understand.
8. repeat step 7 loop until you understand everything.
Now running concurrently with most of these steps, there's two escape/break decisions to be made at almost every point:
a) do I understand paper sufficiently, if yes, then stop reading.
b) is paper actually worth continued reading: if no, then stop reading.
overall, I'd say the vast majority of papers aren't worth reading.
Some are easy and only require ~one read.
A very very very very small amount are both worth reading, and take a substantial amount of time. multiple reads, several weeks/ months to grok. I don't think I'm being hyperbolic when I say the number that justify that probably are less than 10 in your entire lifetime.
If I need to design an experiment, I read the abstract and the methodology, then study the results to see if the methodology is any good. The references in the methodology often lead to other good papers. Here you find discrepancies in what they think they’re doing, what they tell you they’re doing, and what they’re actually doing. Reagents, for example, can be very specialized, and may not actually be available from the vendor they list. It’s a great way to “meet your neighbors”, asking them if they can help you find materials.
These days I mostly read critically, as I’m a practitioner and not a researcher. In this case I read the abstract and introduction. If the introduction isn’t 100% stuff I already know, I go out to the references. This makes critical reading time consuming and causes a fractal explosion of the number of papers in the collection. However, it is also how you find bullshit. Often someone in a review will assert that a reference supports a statement. Lazy or mendacious writers will copy that statement complete with the reference into their own paper. The actual reference may say nothing of the sort. Sometimes, there are multiple levels to this telephone game and each level loses nuance while adding error. When you see a lot of this, you know that there’s a bullshit problem in the domain.
If the author seems prone to BS or plagiarism, the results are somewhat questionable.
If the introduction is fine, I read the results and compare bits to the method. In my field, we look a lot at statistical power and effect sizes. Significance tests are often used incorrectly as a proxy and it pays to understand their limitations.
It’s also important to know how close the result is to the real world (for me). A paper showing reduced mortality, faster recovery, less pain, etc., is much better than one that shows a change in a biochemical marker or cellular protein and RNA content. A study in the species of interest is usually better than one in a model.
Ioannidis is right, most studies are flawed, so it’s best to avoid putting too much stock in any single one. A lot of good research has been done by questioning the accepted wisdom, which usually means one or two studies that may have flaws.
1. Is this paper of interest to me? 2. How will I assimilate the knowledge of this paper into my existing framework?
For (1), the title+abstract helps a lot. Followed by the conclusion, which can indicate whether there is something really to be absorbed.
For (2), I need some grasp of the "related work" - i.e. "how have others tried to solve this problem before this work?" So once I affirm interest, I might scan some of the references. If I'm familiar with them, I know enough to assimilate. Otherwise if the paper describes them adequately for the purpose, I settle for that with a note-to-self to dive into anything in detail.
A third kind of "always on" mode for me is "can I reuse any techniques or results in this work?" and I read more carefully to gain that.
For abstract and structure, my preferred structure (because I was trained like this) is .. in this specific order .. -
1. Introduction - broad sweep description of area to clue people in.
2. Related work - a survey of how people have solved some key thing.
3. Problem statement - What remains unsolved/unaddressed relative to the "related work".
4. Solution - How does this solve it. This includes methods.
5. Validation - How does the author claim that their solution actually solves the problem they set out to tackle. In some cases, this may involve user studies, in others it may be a mathematical result, etc.
This is both my communication framework as well as a knowledge assimilation framework.
edit: I find an abstract that's pretty much like one sentence or so for each of those items in that order easiest to assimilate. However, I admit this wasn't easy to develop initially and I struggled a lot with it before it became normal.
We're investing considerable effort into making it easier for researchers to find and consume scientific literature. Our team is made up of engineers and researchers who have felt the pain points firsthand and are very motivated to design and build solutions that fix them. Our software is free to use, and always will be.
Our search engine uses things like citation intent, citation influence, and figure and table extraction to make filtering through papers easier. We’re also currently prototyping an augmented reading experience that aims to embed contextual information directly into the reading experience so that it's easier to consume and comprehend academic literature. Give it a try: semanticscholar.org.
I'm at my half yearly "I should write my thesis!" cycle and searching for a topic. My idea was reading existing papers and looking in the "future work" sections to find something interesting.
Looking for new ideas? Start hanging around with students/staff from outside your own domain. No better way to get the ball rolling on new perspectives.
In lieu of consistently good science journalism, I try to take a peak at these things in the relevant papers as well. But this requires the additional knowledge of knowing the limits of certain 'methodologies, experimental designs, and statistical analyses' and what you can/can't extrapolate. Which, as somebody outside of formal research, I've only put together piecemeal through reading other criticisms.
Are there any good, comprehensive resources for the limits of different study methods/experimental designs?
1) Title: My research usually starts with a Google Scholar check and going through 4-5 pages and marking everything to Zotero based on title only. I only go through other databases once I exhausted the most interesting GS stuff.
2) Abstract -> Conclusions: If it fits it goes onto the to read stack with a self written 2 line summary of why I included it...I usually start sub-topic stacks at this point
3) Quick browse the methods section and figures/tables etc.
4) Complete read.
If I'm in the beginning of a literature review I also read the sources and marker everything that sounds interesting to add later.
https://simon-frey.com/weeklycspaper/
As I mostly dive into topics regarding distributed systems and backend development the focus will definitely be in that area.
Scientific articles are usually organised in a specific way that makes it easy to get a good understanding of the domain it is providing new knowledge.
What does it mean to read something seriously? We consume content differently according to our own needs. Even if you read something seriously there is always a chance that you didn't read it "seriously" enough.
Try to find meta-analysis to compare multiple methods.
Wikipedias "sources" can range from very accurate and on-topic to complete garbage. They certainly train you to filter out the good from the bad, as well as to consider whether the paper is even relevant to the statement it's being cited for.
Also, there are communities that share a lot of papers. Eg if you're interested in ML (which, if you didn't know, is a CS topic), https://old.reddit.com/r/MachineLearning/ posts a lot of papers.
edit: I just searched for a random CS paper for you. I haven't read it. Are you feeling lucky? https://arxiv.org/abs/1004.2772