This time, Reviewer 1

Published on: February 5, 2015February 5, 2015Author: Amy CollierComment: 0

Rejected.

A paper George Veletsianos, Emily Schneider, and I wrote was rejected from a journal a few months ago. I’m still pretty grumpy about it. What irks me was that the paper was well-received by one of the reviewers, but the Reviewer 1 was having none of it. And it was this person’s review (and the journal editor) that tanked our paper. It was clear that the first reviewer approached qualitative research from a positivistic lens, which led to inappropriate criticisms about sample size and to calls for quantification of the data. I asked a close colleague how someone can review a paper without clearly understanding the research paradigm, and she said, “…some reviewers don’t really try to understand what the authors of the article were trying to accomplish. It’s like all they want to do is criticize the paper for not being what they think it should be, rather than trying to understand what it is. Certainly scholars, even if they don’t use the particular methodologies employed in that study, can understand why those methodologies were employed.”

Sigh.

Of course, we aren’t the only ones who have experienced this. Peer reviews for academic journals are stuff of legend. George introduced me to a Facebook page called Reviewer 2 Must Be Stopped! It’s a space where people share and commiserate about the horrible, and often absurd, reviews they receive.

In some ways, I am comforted by the existence of Reviewer 2 sites, and sites like Shit My Reviewers Say. It’s nice to know that we are not the only ones who have fallen prey to a bad reviewer (even if our “problem” reviewer was actually Reviewer 1). But, on the other hand, reviews like the one below are pervasive and they are somehow tolerated by editors and journals.

What is it about being a reviewer for a paper that makes some people lose their minds and/or their humanity? Is it that academics just feel the need to haze each other, because they’ve been hazed by others? Are we in a self-perpetuating cycle of ugliness?

The issue is exacerbated by peer review processes that obscure reviewers and processes, rather than ones that open dialogue. Blind peer review, specifically, seems to encourage people to unleash their inner beast. We have all seen how anonymous commenting often turns on our favorite news sites (not to mention the oh-so-enjoyable comments on Inside Higher Ed and The Chronicle). Some people have called for peer review processes to be transparent, rather than blind (more later) to help combat this. Yes, I know that plenty of people have written horrific things and proudly published them under their own names, and many named peer reviewers have bullied their colleagues. But by and large, I think people are less willing to be ugly when they have to sign their names to something.

There are other problems with peer review writ large. Fiona Godlee says “at its worst, peer review is seen as expensive, slow, biased, open to abuse, patchy at detecting scientific flaws, and almost useless at detecting fraud or misconduct” (The Ethics of Peer Review, page 59). But I wasn’t really planning to rage against the peer review machine in this post. Instead, I want to highlight some ideas others have had about peer review and ways we can humanize and improve it.

More transparency and onymity

In Re-examining Reviewer Anonymity, Aaron Getz says that many problems could be ameliorated with more transparency in peer review. Anonymous peer review gives reviewers “power without responsibility” (Fiona Godlee, The Ethics of Peer Review, page 65); Getz argues that reviewers should have to “own up” to their reviews. Doing this may not totally solve the issue of people’s civility to one another, but perhaps it will help some people to rethink the truly egotistical, phobic, or biased statements we often see in anonymous reviews. Getz also notes that onymity may improve the quality of reviews, or keep people out of the review process who do not have the expertise to review appropriately. He writes, “It’s easy to write a subpar review; it probably happens every day. It’s not easy, however, to write a subpar review if your name is attached to it. Our desire to maintain our reputation is strong. To illustrate this, I challenge you to tweet or update your Facebook status to this: ‘CBS is hands down the best network. I could watch it all day.'”

More depth and collaboration in peer review

In the past, peer review was not a mechanism for quality control as people treat it now; it was a way editors could get input from colleagues about which papers should be considered for inclusion in a journal. Nowadays, peer review seems to be a way people denigrate each other’s writing and research studies. But what if peer review became a more collaborative process between the authors and the reviewers? What if it became an opportunity to improve papers, not just judge them as inadequate? Studies have shown that proper peer review can improve the overall quality of papers (Godlee). This would require people to give up the attitude, build some trust with each other, and work together to produce good things. It will also require training and guidance from editors, who will need to help people who have not learned how to work with others and give good feedback. I think the Hybrid Pedagogy folks do a great job with this collaborative peer review. Here is more information about their peer review process. I’m sure there are others who make peer review a collaborative and constructive process—please share other examples with me.

More humanity and more love

This may seem like a strange idea, but I trust the great people at Hybrid Pedagogy to know what they are talking about. In Love in the Time of Peer Review, Marisol Brito et al call for love in peer review that humanizes, rather than normalizes. With this as their foundation, peer reviewers for Hybrid Pedagogy seek to be teachers-as-editors and to approach their work with respect and love for authors. What I love even more about their approach is that it does not assume that articles submitted for peer review are even remotely in their final state—it assumes that the articles will emerge through the collaborative process. I love that and I think it’s a much more exciting way to think about the process of writing and publish a paper. It’s not quite Anne Lamott’s shitty first drafts idea, but it’s close. Let’s stop believing that messiness and complexity are problems—let’s stop being uncomfortable with the unfinished.

So I am curious to know what others’ experiences have been with peer review. Tell me about some times it really worked (or not) and why.

Good readings on the topic
JP de Ruiter, How anonymous peer review fails to do its job and damages science

Fiona Godlee, The ethics of peer review

Kathleen Fitzpatrick, Planned Obsolescence

p.s., It looks like our paper may be published later this year. WOOT! While the Reviewer 1 and editor for one journal rejected our paper, a second pair of reviewers with another journal gave us really useful feedback that has helped us to make our paper better. I’ll post a link when it’s up.

Photo Credit: mugley via Compfight cc

Leave a Reply Cancel reply