Peer reviewing: A suggestion

Since completing my PhD, I have served as peer reviewer for several articles. One thing that stands out to me is the fact that doing so is not very time-consuming — basically, you need to read through the article a couple times, then write up a page or two of remarks in response. It’s a process that can easily be completed in an afternoon.

Another thing that stands out to me is the mismatch between how relatively painless the task is and how ridiculously backed up most journals’ review processes tend to be. Part of that surely results from the procrastination that forms the bedrock of the academic life, but I also assume that there are many academics who are doing little to no peer reviewing at all.

And that’s when it hit me: I’m writing this post to procrastinate on reviewing an article!

In all seriousness, though, there’s no reason academics who are actively involved in research shouldn’t be peer reviewing regularly. I’d even suggest that doing as much as one article per month would not be particularly burdensome.

32 thoughts on “Peer reviewing: A suggestion

  1. In addition to the journal back-up problem, I’ve been frustrated as a submitter with the lack of comment I’ve received from reviewers, both in situations of rejection and acceptance. I’ve only really been satisfied with reviewer responses from one of my articles, where they actually made some meaningful suggestions and critiques. In all other cases, one could hardly tell whether reviewers had even read my submission. I can grant that one shouldn’t expect lengthy and detailed commentary from reviewers… that’s why one shares manuscripts with colleagues before the peer review process. But in many cases I think the review stage just isn’t treated seriously enough (at least that’s my impression from being on the submitting side of things.

  2. Whoever asked you to review an “article” was perhaps a little misleading with their terminology there. I did get a breathless sense of the horror of discovery from the italics and exclamation points, though.

  3. I concur with Evan.

    Having been both a managing editor and article writer, I can say that from both sides reviewers often write poor reviews that are also late. Perhaps if reviewing could be included as part of required university service commitments but not part of the tenure process?

  4. I agree that we have an obligation to the profession to review articles, and to do so in a timely and substantive manner. I would go so far as to say that it’s blameworthy if we don’t, but not praiseworthy if we do, which is precisely what demarcates its obligatory nature.

    Having said that, I don’t see how anyone could meet this obligation “easily…in an afternoon” (so Dr. Kotsko), much less “in an hour” (so “bzfgt”). It usually takes me half a day to read a submission and then an entire day to work my way back through it (to ensure I’ve got the argument right) and muster my own argument about its merits relative to publication. That’s only if the submission is well written and relatively brief (under 25 pages); if it’s either longer or poorly written, I have to spend much more time clarifying the author’s argument before I can mount my own. To my mind, that’s what it takes to meet the obligation. If that’s so, and if I’m not unusually slow either at reading or thinking, then it’s farcical to suggest that it could be met in an hour or even an afternoon.

  5. Ben,

    Because I do think think it would be wise to politicize peer-review with the tenure process. We would likely get more of what we have now–even more poorly done reviews. As part of a service requirement, however, might help without unduly politicizing the process. Also, it would be excellent for colleges without tenure.

    Kevin is right.

    Reviews cannot be properly performed in an hour, and only in an afternoon in rarer cases. I’ve seen too many of those reviews (on both the author and editorial end), and they’re not laudable.

  6. Kevin, I honestly think your attention to detail is excessive. If an article is so poorly written that you have to devote that much time simply to parsing what the argument is, then that’s a pretty good hint that it’s not ready for publication, even before you get to the point of analyzing the deeply-buried argument itself. I agree with Evan that it can be frustrating to get what seems to be an arbitrary yea or nay, and I read every article I review at least twice to make sure I’m making a fair assessment — but I honestly don’t think the kind of deep meditation your describing is a good use of your time, nor is it a reasonable expectation of peer reviewers.

  7. Obviously doing it in an hour is not realistic — I’m pretty sure it was thrown out there as a joke. I think that the unrealistic and unnecessary expectation of painstaking labor may be part of what clogs up the process. In writing a review, you have two or three positions you can take (publish, reject, and maybe accept with revisions, etc.). You have to make up your mind which one of those is appropriate, and you have to give a reasonable account of why you think that. I think for an article of moderate length, that’s doable in an afternoon of concentrated work.

  8. @ Mr. Hills: “Taking that long might also be a sign that it’s too far outside of one’s expertise.” Might be, but in this case it’s just not so.

    @ Dr. Kotsko: Perhaps you’re right that I’m attending excessively to detail. But there are two reasons I persist in doing so (beyond the brute fact that I may be unable to do otherwise). First, I don’t think I’m entitled to assess an article until (a) I can reconstruct its argument–plausibly–for myself, and (b) judge its soundness. Reconstructing the argument takes a while even in the best of circumstances, and judging an argument’s soundness even longer, since it requires one to consider what warrants each of the argument’s steps, to what objections each step might be liable, and how the argument’s own premises may have warranted other conclusions. Like I said, doing this takes a while, but I wouldn’t feel warranted in assessing the argument until I had done so.

    Second, you’re undoubtedly right that submissions should be published only if they exhibit a certain degree of clarity, such that one would be warranted in recommending rejection simply because it is difficult to reconstruct the argument. I grant that. But then, I have a hard time imagining writing a reader’s report such as the following: “I cannot recommend publication of this submission because I had a hard time figuring out what it was trying to argue.” That is, unless I could say something further about the (purported) argument, I wouldn’t feel entitled to such a recommendation…but in order to say something about the argument, I need to reconstruct it and assess its goodness, which brings me back to the considerations mentioned above. For me, then, once I’ve started reviewing an essay, it’s hard to put on the brakes at any point prior to a full-scale reconstruction and assessment of the argument, and that takes a while. (But it’s an important part of the job, so I’m not complaining.)

    There may also be other reasons why you and I have different turnaround times. Words may come more quickly to you, for instance, or perhaps there are relevant differences of temperament. Those differences could make a significant difference, in which case I should temper the strength of my original claim. (That is to say, I hereby withdraw the charge of farcicalness–against you, if not against the other poster.)

    One last thing: the original post advances an important claim, and I do not want to detract attention from it by focusing the discussion on turnaround time. If I have done that, I apologize.

  9. OK, in all honesty peer reviewing sounds like a rather enjoyable work. I like editing. What can I say? Anyway, what kind of cred. does one need to this kind of work? And/or how does one get into it?

  10. Usually, one starts getting requests after publishing a first book, as that is the standard way to establish oneself as a scholar. Along with that is membership in the appropriate specialist organizations, who often ask for reviewers, and once you’re considered an established scholar you will be moved up on the reviewer list. Being a reviewer is less about editing and more about evaluation, though any editor will love you for doing their job.

  11. After over a year of waiting, I had a “please resubmit” rejection with a fairly prestigious journal that had two peer reviewers who refused to really comment on the argument but on minor details that were fairly debatable–in fact, they disagreed with each other on many of these points. (I wonder if one of the reviewers saw the other reviewer’s thoughts, because they seemed to be responding to each other.) I asked the editor if she could clarify what exactly the issue is because the comments were not substantive, and my only response was “we agree with the reviewers, but I believe in this article and hope that you resubmit.” I’m not going to waste my time, though I have this great article looking for a place to go.

  12. In light of Dr. Hills’ remarks, as well as some of the foregoing, it might be helpful to hear what folks think a responsible review ought to include. If Drs. Hills and Kotsko were willing to share their thoughts on this, then, I would be grateful.

    (I just discovered, by the way, that the Mr. Hills of my previous comment is in fact Dr. Hills. I intended no disrespect,but I apologize if any was conveyed.)

  13. If an article is so poorly written that you have to devote that much time simply to parsing what the argument is, then that’s a pretty good hint that it’s not ready for publication, even before you get to the point of analyzing the deeply-buried argument itself.

    “not ready for publication” multipliciter loquitur. Does that mean it should be rejected? Does that mean it’s got a good argument (or a good idea) that needs to be made clearer—should you ask for a revision and resubmission? A “major rewrite”? It’s not just an up/down vote.

  14. I think that an article that requires more than a day’s worth of work even to discern the argument deserves an unambiguous “down” vote, but that’s just me. The editors of the Journal of Opaque Prose probably shouldn’t approach me to review anything.

  15. Kevin, We all generally refer to each other by our first names in this context. I appreciate your efforts at showing proper respect, however.

    As far as what a good reader report should include, I think that the guidelines provided by the journal are a good place to start. In general, though, the reader report should include the reasons for the decision you’re making, and that’s going to look different depending on the nature of the article in question.

  16. Kevin,

    Please, the titles are over-kill, though I appreciate it.

    Adam is correct in his comments. After today’s classes, I will post a run-down of the standard editorial/reviewer decisions and what is expected of each. What is expected varies depending on the judgement rendered. As Adam notes, a straight-up rejection is the easiest and fastest.

  17. Why not abandon peer-review practice all together and let the editors decide what they like and don’t like? Is peer-reviewing really a guarantee of the quality of published work? When did it become mainstream and accepted practice in the US? In Russia I’m pretty sure that most of the difficult work of reviewing/commenting is done by the editors who are paid to do (however little), i.e. it’s their freaking job (among other things). I don’t think one can expect academics be diligent with work that is not directly connected to their academic advancement. God, some barely try teaching because it’s not really an important element of their tenure process (or they are tenured already) – do you really expect that appeals to some kind of “academic solidarity” will change the situation? Are we afraid that if peer-review process is dropped, then a whole lot more of shitty essays will appear in journals? It seems that peer-review original purpose was to guarantee the quality of work, but now it’s just a way for editors to pass on work – academics are notorious for willing to work for free, aren’t we?

  18. I could see the benefits of a system where editors were primarily responsible and had the option to consult with experts in particular subfields as necessary. There’s an inertia behind the practice of peer review that would be tough to fight against — it would require shifting a lot of customary practices (including tenure standards), which would be extremely difficult to do given the highly decentralized nature of higher ed in the US (i.e., an individual journal that gave up peer review would lose prestige given that it would no longer “count” for tenure; an individual department or university that stopped caring about peer review would not make that big of a difference because its people would likely still continue to publish primarily in peer-reviewed journals, if only to ensure portability and keep up their repuation, etc.).

  19. Further thought: it’s especially unlikely that peer review would be abandoned given the recent fetishization of “assessment” in higher ed. Even if we know better from experience, there’s no denying that “anonymously vetted by two experts in the field” sounds more rigorous than “approved by the someone who thought it would nicely round out a journal issue.”

  20. Peer review is a better process for at least two reasons. First, it tempers the dictatorial power of editors. Second, it allows for vastly more detailed feedback. How it is implemented is a separate issue.

  21. I was asked to provide some insight into the review process common at American academic journals. The question that I intend to answer is “what is expected of reviewers from the editor’s viewpoint?” I answer this question because the editor, and not the author, has power to set these expectations. I will assume that the editor has performed some basic checks, e.g., fitness to the journal, minimum writing level, etc.

    The expectations differ depending upon the reviewer’s final judgment, and thus I will moderate my responses to each of these:

    acceptance—accept and publish immediately
    conditional acceptance—accept if certain revisions are made. subject to further review
    revise and resubmit—willing to review a heavily revised article
    reject—will not publish or consider again.

    The least is required of the reviewer to outright reject an article. An outright rejection requires basic or large-scale mistakes in the scholarship that rob the article of value, and these are usually quick to point out. Revise and resubmit, and conditional acceptance require the most work. The reviewer should identify why the article is of scholarly value and explicate all the points of revision that would enhance that value. The identification is important, since the suggested revisions could differ a lot if reviewers or editors disagree on its scholarly contribution. Acceptance requires at least as much work, but is almost never given carte blanche. All that said, identifying the scholarly value of an article and its points of weakness can take quite a bit of time, which varies depending on the subject and analytical approach. Reviewers should not editor for grammar, composition, or style, but they should note any problems; the details should be left for the editors. Most journals require forms to be filled-out along with the written evaluation.

    For the record, I have experience editing in the fields of mathematics, philosophy, history, sociology, political science, etc. for academic journals, academic monographs, and professional newsletters.

  22. Professional editors working at for profit journal publishers sets up a power dynamic that can seriously undermine a field. This phenomenon is threatening in some scientific fields. You really don’t want that. I could rant for days about peer review, and do so frequently, but professional editors working for corporate publishers is not the answer.

  23. Hill, I feel like that last dichotomy is a bit grotesque – so the choice is between the ‘god-knows-when-it-will-get-back-with-comments’ system or some clique of professional bullies with corporate ties. The argument that the peer review system can’t be abandoned (for obvious institutional reasons) is not necessary the argument that it shouldn’t be abandoned (for the sake of proliferation of ideas and general philosophical health). Clearly, as Adam points out, the institutional reliance on peer-reviewed journals is the main obstacle to its elimination – but it appears that the only real motivation for doing peer-reviews is a kind of self-motivation: “If I do it quickly, then my colleagues will do it quickly as well” – but is that really going to happen? The tendency is for the review times to get longer not shorter (according to anecdotal evidence from academics). Another issue is the whole idea of “experts” – I have received some requests for reviews for things I know little about. How is the choice made? It’s clearly a part of editorial control – so who is deemed to be an expert could be a subject of abuse. Plus, is it not likely that the real experts would not have any time for peer review?

  24. I think it’s important to focus on something we can surely all agree on, which is that one should not submit an unrevised masters thesis to a journal. Not that I’m saying anyone has recently done that, of course!

  25. I’m a chemist, but the journals to which I typically submit papers have fairly strict deadlines. I’m not ultimately sure how they are enforced, but my experience has been that thing happen in a pretty timely fashion. Sub-fields get pretty small, and word travels fast, even about supposedly confidential things, so perhaps that polices it.

    For professional society published journals, academics serve as editors and they solicit expert reviewers for a paper. These editors, specifically assigned to the subfield of which they are a member, have some flexibility with the reviews and can make judgment calls on borderline cases. This seems to be preferable to the for-profit journals (in my field) which typically have professional editors that, while they have scientific background, typically are not specialists and are not equipped to respond to situations in which a reviewer does a poor job or is incompetent.

    Science specializations, I would imagine, tend to be narrower, so this may be a bigger issue.

  26. This topic just came on on New Apps.

    One idea was mentioned that I like. When the article is published, the names of the reviewers are also published.

Comments are closed.