A methodology for reviewing academic papers
Thursday, October 18, 2007 at 18:03
martian77 in HCCS Adv

Given that this week's paper was generally voted to be a poor example of a paper we tried to come up with a way of reading that might help us to identify this more quickly.

  1. Start with the keywords. If they aren't familiar, it probably isn't the right paper.
  2. Move on to the abstract. Does it sound like it might be useful?
  3. Look at the number and quality of citations this paper has. If it has hardly any, chances are it hasn't set the world alight. Likewise if the only people who have referenced it are the authors, chances are it's not really had a large effect on other people. This isn't rigid - you might get something really useful from an little-known idea - but it could be a good indication.
  4. Then read the conclusions - this was quite an interesting idea. If the conclusions seem woolly and don't refer back to the earlier sections, will it be a bit like that throughout? Also, do the conclusions seem to match the abstract? If not, chances are this isn't a great paper.
  5. Look at the section headings. Again, do they appear to match your interpretation of the abstract. In the paper we reviewed, the abstract suggests that a series of methodologies will be discussed. Are there section headings reflecting those different methodologies?
  6. And when you've read it through, does the paper stand on its own as research? Or do you need to look up other references to understand chunks referred to in passing but central to the argument?
  7. What can you take away from this paper? Could you repeat the processes outlined? If not, it's probably bad.

The introduction should explain the problem space. So identify other similar research, explain what's out there, and where the hole is that this paper is trying to fill. The rest of the paper then goes about filling that hole. It's like a story. The same structure that applies to writing an academic paper can also apply to writing dissertations or theses.

It was suggested we look at other papers to judge a good paper, potentially from outside our field. So the ACM (for example) produce a 'most downloaded paper' chart. Clearly the top papers must have something going for them and would probably be worth a read. Likewise conferences (such as CHI or the British HCI conference) will award a 'Best Paper' for (yes, I know) the best paper at the conference. These would also be worth looking at. And the Journal of Usability Studies (JUS) have a set of submission guidelines that are also worth reading.

 Interestingly white papers are not reviewed. So are more of an opinion rather than a full academic paper and they tend to be commercial. They can still be useful. Need to have a look at whitepapers.com

Article originally appeared on Life on Mars (http://www.martiandaze.net/).
See website for complete article licensing information.