A guest post by Chris Mebane

Momentum is building for SETAC journals to move to a double-blind peer review process. Here, I discuss some of the ethical arguments for double-blind reviewing, practical difficulties, and argue that funding statements and conflicts of interests should not be obscured from reviewers. Hopefully, SETAC authors and readers will join and expand the discussion via comments on IEAM Blog posts.

At the SETAC Publication Advisory Committee (PAC) meeting, held 15 May in Rome, a poll was taken on whether SETAC journals should move to a double-blind peer review process. All members of the committee who were present raised their hands in assent, including the editors-in-chief of the Society’s two journals: Environmental Toxicology and Chemistry (ETC) and Integrated Environmental Assessment and Management (IEAM). While this endorsement does not by itself change practices, it does clearly show that after years of debate, the momentum has strongly shifted towards change. 

The blind-review issue

Peer review vetting is probably the single most important factor distinguishing published journal articles from gray literature. The peer-review imprimatur signals to the readers that the material received impartial, expert reviews prior to being published, and is likely to be credible. Likewise, for many if not most SETAC scientists, earning publications in credible journals is important for building reputations, obtaining support, developing a career, and gaining influence within one’s field. But what if peer reviews were perceived as not being impartial, and instead the playing field was tipped in favor of some authors and against others?

Like the of scientific journals, ETC and IEAM use a single-blind peer review process. The process is “single-blind” because the authors don’t know who the reviewers are, but the reviewers know who the authors are. The handling editor is the go-between, and the identity of the editor is known to all. The single-blind review process is intended to allow frank assessments of manuscripts without the reviewers having to worry about retribution when roles flip. Peer reviewers and peer editors are practicing scientists themselves and today’s critical reviewer or handling editor will be tomorrow’s author hoping to get their manuscript blessed by the reviewers. (These concerns are not entirely hypothetical. I have had highly derogatory and personal writings from disgruntled authors come back around to me over my non-anonymous peer editor role.)

But, how is an author to be assured that their submission will be evaluated solely on the quality of the science and the presentation, rather than preconceptions that reviewers might hold about who did the science? Do papers with “big name” scientists as senior authors get an easier go than unfamiliar authors? Do studies conducted at prestigious research centers get treated with less skepticism than those from say, a teaching university with more limited resources? Might details like nationality, whether they are native English speakers, or gender at least unconsciously affect reviewers’ views toward the work? For instance, overall acceptance rates of manuscripts submitted to ETC are higher for submissions from North America and Europe than from Latin America, Asia-Pacific, or Africa. Does this solely reflect the quality of the science or might implicit bias by reviewers play some role? Why not just obscure the authors’ identity from reviewers to be sure it doesn’t?

These aren’t solely hypothetical concerns. For instance, in computer science, a controlled study showed that from elite institutions were more likely to be accepted to a competitive conference. was shown in single-blind reviews of post-doc fellowships, and early-career researchers in ecology may hold the that peer review and article acceptance rate depends on the prestige of the authors, their nationality (and thus mother tongue), and their institution.

is an early-career African environmental chemist working in China who has published primary research in a variety of well-regarded journals and commentaries in and . In his piece, “ he wonders why peers in his specialized subdiscipline in organic chemistry don’t cite his work. It’s a small field, and anyone working in the field who has ever used a search engine has to know who else is working in the area. According to Sanganyado, his work has never been criticized in related papers. Worse, it has been entirely ignored, even in critical reviews for which his work was directly on point and readily available. He wondered if the reason his work is ignored is because he is an African. I wonder what he wonders when he clicks that “submit” button on a manuscript. The word “submit” has several meanings, not all of which are positive.

Passions are strong among those who believe that traditional peer-review processes are unfair. A published that switching to double-blind reviews increased in an ecology and evolution journal is often repeated, even though it was (the representation of female authors in the journal was similarly increasing before the switch to double-blind). Emily Darling, a marine biologist captured the passion and issues well when she argued that it didn’t matter if data were unsupportive, , and since some people feel the single-blind system is biased against them, it should be changed. In that sense, the choice of how peer review is managed becomes a business decision. For example, found that while the majority of their authors preferred single-blind, a majority of minority and young female researchers supported double-blind peer review and as those are important demographics for the future of their Society, they switched. Similarly, within SETAC, a 2017 survey of authors showed a preference to maintaining single-blind (52%) over double-blind (36%) reviewing with the 12% preferring open reviews. Thus, the PAC overruling the author preferences might reflect a passionate minority outmaneuvering or wearing down a less passionate majority.

All these concerns of implicit bias have given momentum to move to a double-blind peer review of journals. For a true double-blind review process, the identities of the authors, their organization, and other details that could identify the authors, such as where the study was conducted are masked from the reviewers. Usually, this is done by requiring authors to submit two versions of their manuscripts: a regular version seen only by the editors, and a scrubbed version for reviewers that redacts all potentially identifying information.

Ethical and practical concerns with how double-blind review processes would be implemented

Concerns with double-blind review process are two-fold: ethical and practical. The ethical concern has to do whether customary ethics declarations and disclosures would be obscured from reviewers. If an author has a financial interest in something, such as a technique, modeling software, chemical, or product, and their paper under review evaluates that thing, then the author has a financial conflict of interest that ethically needs to be disclosed to readers. Would that conflict be obscured from reviewers in the name of double-blind review?  Likewise, in the science literature, findings often , with examples such as , that , or this other .

Undoubtedly, we all consider ourselves to be objective. Still, the raison d’être for many scientists is to do studies, and most studies need sponsors. Pleased sponsors are probably more likely to go back to the scientist for additional study than are displeased sponsors, and it’s not hard for the scientists to figure out what kind of findings interest the sponsors. The human nature to please one’s sponsor is captured in centuries-old folk idioms such as, “Don’t bite the hand that feeds you” or “know on which side your bread is buttered.”  When science findings follow funding, it doesn’t follow that the findings are invalid. Yet, a purpose of disclosure is to alert readers to the potential for bias, and to encourage open-minded skepticism from readers. Reviewers are the first external readers of a manuscript, and serve as the first line of defense against publishing biased papers. It seems to me that the arguments for disclosing funding sources and competing interests to reviewers are at least as strong as for readers in general. Should this be the case, the fitting folk idiom for double-blind reviewing would be “having the wool pulled over one’s eyes,” which means to deceive by omitting key information. Keeping reviewers in the dark about funding sources or potential competing interests would seem not to be in the best interests of the journals or in rigorous science.

Other arguments against going to double-blind peer review processes are practical: it won’t work as intended and the hassle will turn off authors. Attempts to mask identities often don’t work. Articles published in SETAC journals often follow from similarly-titled presentations at SETAC meetings. Authors usually cite their previous related work which makes them no longer anonymous; certain topics only have a handful of labs or teams actively working on them and trying to obscure the lab or study site is either ineffective or obscures important details from reviewers. The increased work for the editorial office to check for compliance could be costly and slow down the publishing process. Authors may balk at having to go to the trouble of preparing and reconciling redacted and full versions and may walk away. SETAC scientists have a variety of outlets available to publish their works. If publishing in SETAC journals becomes a greater hassle than publishing in non-society, for-profit journals, is it worth the trouble? Finally, if there is a problem with implicit bias, . Reviewers advise, but editors decide. It’s not practical to make the editor blind, and a biased editor could direct an article to allies or assassins for review. For such an editor, a double-blind review policy would only provide a falsely reassuring veneer. Thus, to skeptics, double-blind reviews sound like a disruptive, incomplete, and dubious solution to an ambiguous problem.

Double-blurred: A compromise between concept and practicality

The compromise approach to double-blind peer review endorsed by the PAC could skirt some of the practical concerns with the practice. The idea is to follow a soft, mandatory double-blind approach adopted by some other journals. In this approach, the extent of blinding or merely blurring is largely left to the authors. Authors would be required to submit an anonymous cover letter and two versions of their title pages – one with title and abstract only that would go to the reviewers, and conventional title page seen only by the editor. As anticipated by the PAC, the acknowledgements would be redacted, although I argue that that should be limited to redacting names of individuals, but not funding sources, and conflict of interest statements should be framed without naming names. Authors would not be prohibited from self-citations, describing study locations, nor required to scrub lab details or other potentially identifying details. While at least the identity of the research group could likely be easily deduced, the concept is to emphasize to reviewers to focus on the substance and merits of the work, rather than the people.

As explained by the editors of a journal giving it a try (Global Ecology and Biogeography [GEB]), “soft mandatory double-blind” is intended to pursue the perceived benefits of double-blind review, yet trying not to overburden authors and the editorial office.  According to the , “…‘soft’ is recognizing that it is nearly impossible to create 100% effective blinding, and we are not even going to try. Policing the degree of blinding would involve somebody reading the entire paper to remove any hints in citation patterns, etc. We feel that our approach is capturing 90% of the benefit without burdening authors or the peer-review system with unwarranted (and probably inefficient) effort. Authors will now submit two title pages, one with author names and institutions and one without (i.e., only title and keywords).”

Why just peer review? What about a more comprehensive look at SETAC publishing practices?

Science publishing is in a state of flux, and peer review is but one of many science integrity issues swirling around publishing practices. A key issue is whether science journals should continue to publish 20^th century style, , in which the behind the concise, highly summarized table or figure results are not available to readers. This would be its own discussion, among others.

Chris Mebane ([email protected]) is a long-time SETAC member who variously has held roles of author, reviewer, or editor.

Update: Two article links were missing in an earlier version of this post. They have now been corrected, as highlighted below. The links can be found in the section Ethical and practical concerns with how double-blind review processes would be implemented:

“Likewise, in the science literature, findings often , with examples such as , that , or this other .”

“Finally, if there is a problem with implicit bias, .”