Last week, I tweeted about a toolI use when I review papers I’ve reviewed before, especially at a different journal (basically, you can upload two pdfs and it will highlight deletions and additions in each one). This prompted a discussion about whether one should ever agree to review a paper for the second time at a different journal. I realized that my thoughts on this did not fit into tweet-sized text, so I decided to blog about the pros and cons of having the same reviewer review a paper at more than one journal. Ultimately, my recommendation for what to do if you’re in such a situation is to ask the editor what she or he prefers, but I think the pros and cons are worth thinking about for their own sake.

First, it’s worth noting that if you get the same person twice, she or he is likely to be quite knowledgeable in your area of research. Otherwise, it’s unlikely that two or more editors would independently think of him or her. You’re certainly not getting that third-year PhD student a second time! That fact on its own should benefit you if your paper is good (which I would hope you think is the case), as someone who is an expert may be better able to appreciate your contribution than someone who is not. It can be bad if you’re challenging a long-standing orthodoxy in your field. But in that case, it’s hard to see how getting a different draw would be that much more helpful – by definition, orthodoxy is something “generally accepted”, so your chances of getting a more favorable reviewer are unlikely to be muchhigher with a new draw. If your paper is challenging a particular person’s work, then it may be a good idea to note that person’s potential conflict of interest in a cover letter when you first submit the paper (note that I have no idea how often this is actually done).

Second, you may be concerned that someone who has reviewed your paper at a different journal and thus has seen it rejected (or even played a hand in that rejection) now has preconceived notions about its quality and is less likely to be favorable than a new reviewer. Here, it’s hard to know for sure what the objective truth is, but I’ll offer some thoughts. Papers get rejected for many reasons. Sometimes the reviewers think it’s a “crowded literature” and that your contribution is too marginal for publication in that particular journal. In that case, getting the same reviewer at a similarly-ranked journal is unlikely to be good news, unless you failed to spell out your contribution clearly the first time around. But getting the same reviewer at a lower-ranked journal is not necessarily bad – what’s not good enough for AER could be good enough for Journal of Development Economics.

If your paper wasn’t rejected for not making enough of a contribution, it was probably rejected for not being convincing enough in one way or another. Maybe you didn’t cluster your standard errors or maybe your null finding is not precise enough. Maybe the reviewer doesn’t believe your instrument is valid. Maybe you don’t have some robustness check. Maybe it’s just lots of little things that together make the reviewer think your paper is not salvageable. Here, there’s usually something you can do to improve the paper. Yes, you might not be able to find a different instrument, but maybe you can think of some ways to indirectly probe its validity. You can try that robustness check. You can add a footnote explaining why doing X doesn’t make sense in your case. All these should make the reviewer view your paper more favorably next time around. By contrast, suppose you get a different reviewer. If you didn’t do any of the things the previous reviewer suggested, the new reviewer might well have very similar concerns (my experience is that the major reviewer comments are pretty correlated!). If you did do something to address the previous reviewers’ concerns, the new reviewer is surely to think of some new ones to bring to your attention (the correlation coefficient is not 1!). That may mean a more rigorous review for the paper as a whole, but it will also make your life a whole lot harder.

Note that I am not suggesting you address literally everything a reviewer brought up, just in case you get him or her again. My strategy is to try to address things that are likely to also be brought up by different reviewers in the future (I personally have never gotten a referee report that did not contain a singleuseful suggestion) and that are not too difficult to do.

It’s also worth noting that the reviewer you’re getting again may not have recommended rejection in the first place. In that case, getting him or her again is good news, especially because presumably you changed something in response to their comments (see point above).

I’ve reviewed the same papers multiple times (3 times is my record), and I’ve had a few of my papers reviewed by the same referee at different journals (incidentally, here the record is also 3 times getting the same reviewer). Generally, these have not been negative experiences. The negative experiences I have heard about both involved the authors changing something in between submissions and the reviewer literally copy-pasting the same report. That is unfortunate indeed. But I think that is a failure of the individual reviewer and the reason I recommend all reviewers in such situations use the toolabove.

I think the bottom line is that it’s an empirical questions whether or not it’s good to have the same reviewer look at your paper again or not (someone should do an RCT on this). If most reviewers are reasonable and diligent people, then it’s not a problem. If most reviewers are lazy and vindictive, then it is. I do think we live in the former world, and (on average) I’m optimistic when I think about getting the same reviewer again.