Under Review: Refereeing Articles

I have been asked by some folks to spew about reviewing--not when to say yes or no; not rules for reviewing--but what are one's standards?  Do one's standards depend on the journal?  Yes.

In reviewing stuff, there are standards that are general and those that are specific or varying.

For every journal article, I expect to varying degrees:

• a relatively clear argument,
• organized logically,
• situated within existing knowledge. That is, the author(s) has read and understood the relevant literature by using the existing works to define the question, to explain how the question has been answered, why those answers are insufficient.
• For me, endless lit reviews are not desirable.  The lit review does not even have to be caleld a lit review.  It does not have to be complete, but it does have to be sufficient.  It does not have to cite everything written on a topic, but it should cite the stuff that a reasonable scholar in a similar position would be aware of that speaks directly to the piece.
• One should not demand an author cite one's own work--that is worse than cheesy.  On the other hand, if you have written the definitive work on topic x, and the person is clearly addressing topic x in a way that needs to be aware of your work, then tell the editors.
• One should never expect an author to have read something that is not in press.  Just because you have a friend or network that has made you aware of a forthcoming book or a wonderful working paper, it is not reasonable to expect unpublished work to be well known.  Unless it is an old unpublished work that has been cited previous and often.
• explains the method.  This is where qualitative work (which is mostly what I do) needs to learn from quant--be explicit about assumptions, choices, etc.  Not all quantitative work is that clear either, but there is a greater expectation on those folks.
• presents the research in ways that relate to the argument.  
• discusses some of the findings.  Just the most relevant stuff.  Not statistical result or quirk of the case study needs heaps of elaboration.
• concludes by tying things together and suggesting implications and broader relevance.

 What is more specific from journal to journal?  Two things: the scope of the journal and how much of a contribution the piece makes.  If a journal is focused on security issues, but the article is focused on stuff that is about security, well, I should not have been reading it.  It should have been a desk reject.  Scope rarely comes into play.  But contribution really is the key.  How much of a contribution is the piece?  Is the contribution significant enough to warrant interest beyond those interested in that exact question?  The wider the perceived significance, the more visible/selective the journal it should be in.  If I read a piece submitted for a major journal, and the contribution is slight, I will be more likely to reject it.  If I read a piece for a more specialized journal, I will expect a different kind of contribution--less about theories that should travel to other domains and more about improving our understanding of a specific causal mechanism or a specific case.

To be honest, most of the reviews I do hinge much more so on the general criteria--is this a good piece of research and less on the question of contribution.

I am sure I am overlooking stuff, and I will be more self-conscious the next time I review a piece.  Which is tomorrow since I am late with one.  Oy.