Peer review: What’s the alternative?

In what is increasingly becoming a tired schtick, Rebecca Schumann rehearses the most inflammatory cliches about academic practices — in this case, peer review. David Perry’s response reflects my own experience: I’ve had a generally good experience, but I’ve heard about friends who had a much harder time.

Schumann suggests that anyone submitting to a journal should be required to do a “constructive,” timely review of an article for said journal before their piece will be considered. The gap here is that the same editors who let people get away with harsh reviews are going to be in charge of assessing the required reviews — why would we expect anything different? It’s also striking that the solution to the badness of peer review is to get more people to do peer review (especially in the case of the “crowd-sourced peer review” solution she also discusses). Yes, I agree: the solution to the problems of the system is a good version of the system without the problems.

I would like to gently suggest that there are factors other than the badness of reviews themselves that contribute to the bad reputation of peer review. First, it makes sense that an increasingly high-stakes “publish or perish” mentality would lead to a higher volume of lower-quality material. Many articles that I’ve reviewed struck me as simply needing more incubation time — including the time and distance necessary to say, “Okay, so I’ve done all this exposition… but what am I trying to say here?” Students who let the deadline sneak up on them are not the only ones turning in “papers in search of a thesis.” And in some cases, I honestly think the editor should have simply rejected the articles out of hand without involving me. It’s not the author’s fault, but I can understand being irritable when you’ve volunteered your free labor and your input isn’t actually warranted.

Second, the high stakes involved amplify people’s usual defensiveness of their own work. This may lead to an overdiagnosis of “harshness” of tone and an exaggerated suspicion of the reviewer’s motives — isn’t it a little convenient that peer reviewers are so often presented as promoting their own work and viewpoints? We’re all willing to call bullshit when blog commenters say that we’re just afraid of how right their views are and can’t tolerate dissent. There’s always going to be something uncomfortable about getting criticism of one’s work from an anonymous source — I mean, I was sweating bullets when a very close friend was reading my draft of Creepiness, and I can’t imagine how nervous I would have been if it were a total stranger — and while some reviewers may come across as unduly harsh, maybe you wouldn’t read it that way if you knew them personally. The disadvantages of anonymity go both ways.

So here’s my proposal: get rid of the anonymity. Turn the process into a genuine dialogue between peers rather than a high-stakes, up-or-down evaluation by some random person. You could even require people to do some peer review on their own beforehand and submit a signed statement from one or two readers along with the piece. To make sure people aren’t constantly relying on the same two people, you could make it a norm for CVs that your initial reviewers are included in the listing for any peer-reviewed work. If you want an independent assessment, have a named peer reviewer respond to the full package.

Yes, there is potential for abuse with this system, in that it could amplify “old boys network”-type effects — but my #slatepitch position is that many of the cruellest and most pointlessly time-consuming aspects come from pretending that the profession is an impersonal meritocracy and feigning ignorance of the importance of reputation and social networks.