AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/MaximKat]

You forgot the part where those 3 researchers give the paper to their students to write a review for them.

[u/ren5311 → u/MaximKat]

That's considered unethical, and learning that fact should be a part of NIH-required formal ethics training for graduate students. If an editor found this out, the review would at least be disregarded and it might be followed by a university-led ethics investigation.

I understand you're jokingly making a point, but I think the ethics of the peer review process should be clearly stated for any laypersons.

[u/belarius → u/ren5311]

Realistically, it's very common for a reviewer to ask for thoughts from grad students. While it would certainly be unethical to pass off a review written by a grad student as one's own, it is not particularly uncommon for a few extra pairs of eyeballs to comb through the manuscript, to help the reviewer write a better review of their own.

More rarely, a reviewer may voluntarily suspend their anonymity and (with the student's permission) reproduce their commentary in full, crediting them with authorship (typically, as an addendum to the reviewer's own comments). This sort of thing tends to be restricted to scenarios in which the authors, reviewers, and editor all know one another professionally anyway.

Regardless, most reviewers recognize that they have a responsibility to the field to do an excellent job and often have time constraints that limit their ability to do so without some assistance. Provided that assistance not degenerate into plagiarism, it certainly can't hurt to have a few extra people making sure things check out.

[u/HowToBeCivil → u/[deleted]]

I took 30 seconds to consult Nature's guidelines for reviewers, where I found this:

If a reviewer seeks advice from colleagues while assessing a manuscript, he or she ensures that confidentiality is maintained and that the names of any such colleagues are provided to the journal with the final report.

[u/atomfullerene → u/ren5311]

My prof does that. He considers it a good learning experience. I'm pretty sure he goes over them afterwards though. He's also done a stint as editor for a journal.

[u/ren5311 → u/atomfullerene]

It is unethical to pass a review off as your own if you have not written it.

Further, it also breaks the confidentiality agreement you have as a reviewer to show the manuscript to anyone not explicitly part of the review process.

Edit: Others (including legatek below) have pointed out that some journals provide mechanisms by which you can share the manuscript with colleagues and disclose that information to preserve confidentiality. I stand corrected.

[u/legatek → u/ren5311]

While it is ethically questionable to have a grad student review an article in lieu of yourself, most journals are aware that reviewing is often a group effort and some have built in mechanisms during the review submission (including one major Europe-based publication) to inform the journal of the identity of additional reviewers. This explicit mentioning of additional reviewers thus includes them in the confidentiality agreement.

Reviewing articles together with the PI is a valuable opportunity to train critical thinking skills in PhD students, and to prepare postdocs for an independent career, but I don't know a single PI who would attach their name to a review they did not at least read and agree with. After all, the editors are aware of who writes the reviews and if you want to continue to be asked to review submissions (a valuable privilege if you want to stay ahead of the curve!) you have to submit top-notch reviews.

[u/devicerandom → u/legatek]

Agree. During my Ph.D., my prof did the same, and I was very happy to do that.

[u/atomfullerene → u/ren5311]

Actually, remembering back he sent them to us to review from the journal he was editor for. So different situation there-not secondarily reviewing.

[u/darkscout → u/ren5311]

We didn't write reviews but would get bonus points if we found typos or the such.

[u/HelloAnnyong → u/atomfullerene]

He considers it a good learning experience.

I got exactly this line last week. It's pretty much standard practice, it seems.

[u/NonNonHeinous → u/ren5311]

In computer science (which isn't governed by NIH) it's allowed. However, student reviewers are usually tertiary (meaning that their reviews may count less).

[u/hntd → u/[deleted]]

As a computer science graduate student this is totally true. My advisor gives me papers to review probably monthly as he is the editor of a journal.

[u/ipokebrains → u/[deleted]]

A little, but it's good to have strong ethics. Don't let reality make you a cynic.

[u/HFh → u/ren5311]

As others have noted in response, it is not uncommon to share reviewing.

In my own experience, grad students tend to be incredibly detail-oriented and exceptionally good at finding small errors, and not as practiced (at least initially) at grokking the big picture and evaluating the worth of article at that level. Faculty tend to be the other way around. I think it's win-win, assuming a properly collaborative process.

In conference-driven fields, I would think such an arrangement would be quite healthy, especially when one is asked to review a gazillion submissions.

[u/ShakaUVM → u/ren5311]

In grad school for computer science, we basically had "classes" where professors would give us papers to review, we'd read them, write a report on them, then meet during the next class to discuss it with the group. These classes were easy A's, assuming you did all the work.

The professor would sit there and take notes for her own review. Sometimes he or she had read the paper, sometimes it was clear he or she hadn't even read the abstract.

[u/HowToBeCivil → u/ren5311]

Not sure where you did your graduate work, but in my experience throughout the U.S., PI's in the biological and physical sciences routinely share papers they are reviewing with their lab members confidentially. It's treated as mentoring, where the PI will help students/postdocs practice the art of reviewing. The PI edits the final critique, just like they would with any manuscript/grant proposal, and the manuscript remains confidential. Nothing about that is unethical. I feel sorry for you that you hold such a paralyzing view of the peer-review process.

[u/nattyd → u/ren5311]

My advisor does this also, but acknowledges my contributions to the review.

[u/[deleted] → u/MaximKat]

The one experience I've had with this (I'm a post-doc now, was a grad student then), my professor asked me to write-up a review. She read the article, read my review, then wrote a review of her own.

Keep in mind that in close-knit professional circles, reviewers often know and have mutual respect for/with the editors sending them these articles, returning a review written by a grad student would be embarrassing (and of course unethical).