How to write a great research paper [pdf] | Hacker News Reader

How to write a great research paper [pdf]

research.microsoft.com

251 pointssiromoney62 comments

Perhaps no one is going to say it, so I'll say it: The nonchalant font makes these slides easy and fun to read.

Also, for who are not in the know, Simon Peyton Jones is a well respected computer scientist, mostly known for his work on developing the functional programming language Haskell (together with others ofcourse), Haskell research is funded for a large part by Microsoft I think.

I think that at some point, he used to use Comic Sans but everyone got a aneurysm so he switched.

Ha, I remember seeing him talk at a conference in 1991 and it was hand-written transparencies on an overhead projector. Comic Sans is a relief. :-)

Also, to add what others have said, he is one of the best speakers I've ever been lucky enough to see in person: SPJ, Philip Wadler, Guy Steele... good company.

Yep, he once asked if there was anything he should change about this talk - I said get rid of the Comic Sans, but keep the casual feel. This was in 2010. You know that a is talk finely tuned when the only thing someone can think to change is the font!

So now he's using a font that makes it look like he's had an aneurysm? Brilliant!

(I probably shouldn't be making any handwriting/typography comments.)

He had an aneurysm, which caused him to choose comic sans. He had a second aneurysm which caused him to choose that thing he's got on there now.

It's also great advice. I especially liked the part at the end about using simple and direct language and the active voice. This alone makes a paper much easier to digest even if the subject matter is hard.

Precisely because he's that respected, I was surprised at the font choice, which I disliked (it just seems to mean extra mental effort for me over something I'm more used to).

He has said elsewhere, in response to questions about his choice of fonts:

  This is a very funny question, why I use Comic Sans. So. All my talks use
  Comic Sans, and I frequently see little remarks, “Simon Peyton Jones, great
  talk about Haskell, but why did he use Comic Sans?” But nobody’s ever been 
  able to tell me what’s wrong with it. I think it’s a nice, legible font, I 
  like it. So until someone explains to me — I understand that it’s meant to be 
  naff, but I don't care about naff stuff, it’s meant to be able to read it. So 
  if you’ve got some rational reasons why I should not, then I’ll listen to them.
  But just being unfashionable, I don’t care.

he also has enough research experience that it's not taken as unprofessional

This feels like good advice not just for writing a paper but for reading one as well. Explicitly thinking about the structure of a paper certainly helps me get through it. And if something is missing--say, good examples--I know to look elsewhere for them before continuing.

Sometimes, finding a slide deck the author of the paper used is more useful than the paper itself just because they have more room for examples! I've found this particularly true for very dense theory papers where I only half-understand the relevant mathematics.

Talk version: https://www.youtube.com/watch?v=g3dkRsTqdDA

Highly recommended. Simon is a very good speaker.

I am a strong advocate of this method.

However, I have to issue a statement of warning that when an idea doesn't pan out like written out, people will go to great lengths to fulfill their agenda (e.g., put undue pressure on their students to come up with the "correct" result). For computer science and mathematics, it may be more difficult to fudge results. But in experimental work or data analysis, I've seen students suddenly come up with a solution that fits their advisor's expectation.

There has to be some mechanism to guard against this; the most common answer is the moral character and integrity of the scientist, but psychologists and economists will quickly point out that there are gray areas where people are willing to cheat just enough to get their reward while preserving their conscience.

This link came together at a good time because I usually get bogged down in the research stage. I always over research and then I lose hope and confidence in myself and mankind.

Pretty cool. Hopefully some of the "lol acadmics" folks on HN will see how good research practices are very similar to startups (and yes that would be the biggest asset of any good academic you hire) :)

yep. getting stuff done in academia today is quite similar to startups, s/investors/government-agencies, s/grants/investment-rounds, s/papers/pitches, s/prototypes/MVPs, s/hiring-students-and-postdocs/hiring-employees, etc.

I'm not the best at writing the papers, but this is precisely how I approach giving talks. I take my idea, submit the talk, then as I build the talk I understand my idea better and better and finally, when I am on stage, explaining my idea using the structure I developed while preparing the talk, I have moments of inspiration that I never get elsewhere.

I have, mid-sentence, come to understand truths about my subject that had evaded detection for months prior. My talks thrive, but now I need to transition to papers. I'm going to try to put this method into practice for myself.

I have to say that this sounds like a good idea to me.

A few months ago I found a way to reduce a particular AI algorithm's memory usage from O(n) to O(1), but I've been bogged-down in irrelevant details when trying to implement this improved version.

I think I'll take SPJ's advice and write up a theoretical/motivational paper describing the improvement, and leave the actual implementation for afterwards.

I've been holding off writing up since I'm an amateur, and AI has a reputation for attracting cranks with hand-wavey conjectures :S

I wish he also recommended that every research paper needs to have a date on it. It seems to be the golden rule never to specify any date on research papers, which makes it really hard to evaluate the paper based on when it was published.

I agree, all papers should specify the date. Unfortunately, when submitting to journals or conferences, the author often doesn't have a choice in the matter, as papers have to follow a given template.

This works well for giving talks as well. I have often signed up to give talks on subjects I wanted to learn more about. In fact I believe the best time to give a talk is when you're still learning a topic, as you're more enthusiastic and have any easier time empathizing with those new to the subject.

Idea - Write manual - Make program

A manual isn't particularly useful on its own, since it's full of implementation details for one particular widget. SPJ is emphasising that details and descriptions of implementations should be avoided in favour of ideas and intuitions.

A purely theoretical paper can still be useful. It's even more useful if it's followed up by an experimental paper demonstrating or refuting the idea.

For example, it's silly to write a manual "How to use Google", then make Google. It's very sensible to write a paper "A proposal for improving Web search with mutual-popularity measures", then try to make Google. If Google succeeds, we can write "Mutual-popularity measures and Web search: large-scale experimental results". If it fails, we can write "Scaling issues on the Web: impacts on mutual-popularity algorithms".

It would indeed be silly, since Google is not a program.

Google’s search engine, however, is, and it would be perfectly sensible to first write a manual for a search engine before implementing it.

From http://aosabook.org/en/bdb.html:

Skilled programmers use different techniques to this end: some write a first version and throw it away, some write extensive manual pages or design documents, others fill out a code template where every requirement is identified and assigned to a specific function or comment. For example, in Berkeley DB, we created a complete set of Unix-style manual pages for the access methods and underlying components before writing any code.

Which is the 3rd circle of hell.

Breaking with this sort of practice is exactly why agile development and it's ilk were born.

While writing a formal specification can be useful, it is rarely as useful as defining the problem and placing your solution in the context of other possible solutions.

I've been lucky enough to attend one of Peyton Jone's talks in person. He is not suggesting to "write a formal spec" before you do research. In contrast, his attempt is to be agile: keep going back and forth in the paper writing as you do research. Only then you will see what's missing.

Essentially, what he is suggesting is actually "agile academic research", compared to the "waterfall academic research" (do stuff, then write-up a paper with what you've got...)

I'm not sure that I agree that actually writing the paper/talk first develops the best output, but I definitely agree that conceiving the idea first is a great route to take.

I've always had a moderate fear of public speaking--I've heard that many do--but in the last six months, I've put myself "out there" because I had interesting ideas that I wanted to share. I went from having given one talk ever in my career to doing six in the last six months. It was scary at first, but I'm getting better... and my ideas have been well received.

The most important point I'd like to underscore here (the same one that the PDF mentions) is that your idea doesn't need to be perfect and infallible to be worthwhile. It just needs to be interesting enough (to you, or to others) to spark a conversation. Always remember that journals and conferences won't accept your material if they don't think it's any good!

Video: http://sms.cam.ac.uk/media/1464870

Meh, compare that to Bret Victor's version which is way better IMO

http://worrydream.com/#!/ScientificCommunicationAsSequential...

I think you are seeing a dichotomy where there is none. Simon Peyton Jones is advocating a process for how to go about doing research and communicating your ideas. Bret Victor is advocating a method for better communicating ideas. One could easy do both.

I actually apply this to my hacking these days. The statement you often realize that problems are more interesting once you sit down and analyse them I have found to be particularly true.

When I'm faced with a design problem I write a "white paper" setting out the problem, framing the desired properties of any solution then as systematically as possible setting out the alternatives and justifying why a particular solution has been chosen.

I usually find that in going through this process I am forced to change my initial design decisions, often ones that seemed obviously, even trivially, correct to begin with.

I want to point out slide 21: No “rest of this paper is...”

This practice drives me crazy, because it's essentially just an outline, which is useless. It's a computer science research paper, I already know the outline! His advice to not have that paragraph, but instead put in forward references to the relevant sections in the introduction is spot-on.

Now if you'll excuse me, I have to get back to reviewing papers which did not take this advice...

I hate that practice as well, yet have still been directed to do it by editors.

also the constant use of "the proposed method/approach/technique/algorithm/etc"

Another way to look at this, which is how I learned it in grad school, is that your goal is to tell a Story that others Believe. The Machievellian corollary is: whether the story itself represents reality is less important than whether you have people who believe in your story. When people believe in your story, you have power. You use that power to get funding, which gives you runway to work on real problems.

I like:

Idea --> Write Paper --> Do Research

It gives me something I can do when I can't work on the research itself (e.g., during meetings, in between interruptions, at home, etc.). And it helps me anticipate obvious questions that might detract from my result. A slide that looks like it doesn't contribute to the impact of the talk might indicate work that doesn't really contribute to the result.

i don't know how that'd work - a lot of my writing starts from indecipherable technical details from research. From there i unravel their meaning to explain the connection between the high level intuition and low level workings. Writing the otherway seems like you'd constantly lose track of what's important.

I seem to get the active/passive voice criticism a lot. It comes from both sides of the table. Do you have references on what to prefer? They can be opinionated, but preferably I would like to see something backed up by a reading comprehension study. Thanks in advance.

Is the suggestion of putting "related work" at the end non-standard? As an occasional reader of CS papers I generally remember this part appearing after the introduction, and feeling quite sapped once I'd got through it and the references.

I've always seen "related work" appearing immediately following the introduction. The layout I see most often is:

* Abstract: This is why you should read this paper.

* Introduction: What the problem is that we're trying to solve and a claim to having solved it better.

* Related work: How other people have solved the same problem (and why we don't like their solutions).

* One or more sections go here here describing how we solved the problem.

* Experimental results: Look at the awesome results our solution provides. (Not always applicable depending on the paper.)

* Future work: Here's some ideas we haven't worked out fully -- they may appear in future papers but we're putting them here so we can claim to have published them first if someone else writes a paper about them before we do.

* Conclusions: Everything in the Introduction, except written in the past tense.

* References: Everything your potential referees have written which might conceivably be relevant.

In PL and systems, related work almost always comes at the end. HCI is the only field I know where related work is front loaded. Pandering to the PC in related work is also bad form.

That said, a quick glance over the PC/ERC to ensure you cite-and-compare is a good idea. It's always easier to answer questions in your paper than the rebuttal. Or next year's submission.

That is an incredibly biased way of handling related work. Doing it right means being blind wrt the PC, if the conference isn't run by dysfunctional egoists and you cover related work in good faith, the reviewer won't dock too many points for missing something and they'll fill in your missing information in the review.

In AI, related work is usually at the beginning, but sometimes at the end. A common form is to set up the problem in the intro (section 1), then use the related-work section (section 2) to survey existing solutions to the problem and explain 1) whether you build on any of them, and 2) what's still left to do. It's sort of like an extended motivation: the intro motivates the problem, and the related-work section motivates your choice of jumping-off point.

We have to mention previous work in the intro and body of the paper to establish context, but the related work at the end is more of comprehensive since the paper has already been fully stated. If your paper is low on context, the reviewer might flip to related work after the intro to fill in the gaps, but you are in negative points territory at that point.

Interesting. Most of the papers I read are mathematics.

I have seen both ways (in computer science/systems). I buy the reasons SPJ says in the talk, but if the problem you are working on is too crowded that the first thing that comes to a reader's mind is -- "How is this different from the N other proposals I've read in the past?" -- the related work helps set the context.

It's a little non-standard, and some would argue that it deviates from the philosophy of "standing on the shoulders of giants", but I think it's acceptable.

Note that you should still "cite relevant work in passing".

It's probably non-standard, but I totally buy SPJ's suggestion. Often when I read the related work section at the beginning I'm confused: I don't understand the paper I'm reading yet, let alone any other related work.

He specifically explains why he suggests this different structure on page 24.

Basically, if you put a huge discussion of related work at the beginning you'll either make your reader feel tired or stupid or both.

The link he gives at the end, Advice on research and writing, is dead.

I believe the updated link would be http://http://www.cs.cmu.edu/~mleone/how-to.html

Reminds me of http://www.allthingsdistributed.com/2006/11/working_backward...

> The Simon PJ question: is there any typewriter font?

On the other hand, a paper that is 80% typewriter font (or 80% mathematics) is a paper for putting down and leaving alone.

BMC journals are somewhat going in that direction strongly suggesting to put the methodology after both results and conclusions.

> Use CVS to support collaboration

o_O

I made the same face, but presumably he means a CVS, such as a DCVS, such as Git.

The generic term is “VCS” (Version Control System). CVS is a specific implementation.

I think he literally meant the CVS given the age of Simon... lol

But how do we get the idea? :)

He actually addresses this question. His advice is: start with any idea, even a bad one, and write about it. You're ideas will develop (and new ones will emerge) as you go. (Thoughtful) writing is the discovery mechanism.

Excellent

Reply on news.ycombinator.com