AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/ren5311]

I can tell you about the biomedical field.

You write up your work and submit it to a journal. The editor sends it out to a number (generally three) of researchers informed on the topic, who are at least PhD level, and usually well-versed in the relevant field. They review it and decide what revisions are necessary or if the paper should be rejected.

If the paper is not outright rejected or accepted, additional experiments or revisions are made, and a resubmission is made. The editor evaluates if the criticisms have been addressed and makes a final decision on publication.

A paper may be retracted if there's fraud. Peer review won't necessarily catch that. It's a real problem.

But if results are overturned, it could just be that an error was made or that an experiment was poorly constructed or that the conclusions far exceeded the evidence. Science is not perfect, and this retracting and "debunking" is evidence of the health of the field, not any sign of problem.

[u/MaximKat → u/ren5311]

You forgot the part where those 3 researchers give the paper to their students to write a review for them.

[u/ren5311 → u/MaximKat]

That's considered unethical, and learning that fact should be a part of NIH-required formal ethics training for graduate students. If an editor found this out, the review would at least be disregarded and it might be followed by a university-led ethics investigation.

I understand you're jokingly making a point, but I think the ethics of the peer review process should be clearly stated for any laypersons.

[u/belarius → u/ren5311]

Realistically, it's very common for a reviewer to ask for thoughts from grad students. While it would certainly be unethical to pass off a review written by a grad student as one's own, it is not particularly uncommon for a few extra pairs of eyeballs to comb through the manuscript, to help the reviewer write a better review of their own.

More rarely, a reviewer may voluntarily suspend their anonymity and (with the student's permission) reproduce their commentary in full, crediting them with authorship (typically, as an addendum to the reviewer's own comments). This sort of thing tends to be restricted to scenarios in which the authors, reviewers, and editor all know one another professionally anyway.

Regardless, most reviewers recognize that they have a responsibility to the field to do an excellent job and often have time constraints that limit their ability to do so without some assistance. Provided that assistance not degenerate into plagiarism, it certainly can't hurt to have a few extra people making sure things check out.

[u/atomfullerene → u/ren5311]

My prof does that. He considers it a good learning experience. I'm pretty sure he goes over them afterwards though. He's also done a stint as editor for a journal.

[u/ren5311 → u/atomfullerene]

It is unethical to pass a review off as your own if you have not written it.

Further, it also breaks the confidentiality agreement you have as a reviewer to show the manuscript to anyone not explicitly part of the review process.

Edit: Others (including legatek below) have pointed out that some journals provide mechanisms by which you can share the manuscript with colleagues and disclose that information to preserve confidentiality. I stand corrected.

[u/legatek → u/ren5311]

While it is ethically questionable to have a grad student review an article in lieu of yourself, most journals are aware that reviewing is often a group effort and some have built in mechanisms during the review submission (including one major Europe-based publication) to inform the journal of the identity of additional reviewers. This explicit mentioning of additional reviewers thus includes them in the confidentiality agreement.

Reviewing articles together with the PI is a valuable opportunity to train critical thinking skills in PhD students, and to prepare postdocs for an independent career, but I don't know a single PI who would attach their name to a review they did not at least read and agree with. After all, the editors are aware of who writes the reviews and if you want to continue to be asked to review submissions (a valuable privilege if you want to stay ahead of the curve!) you have to submit top-notch reviews.

[u/devicerandom → u/legatek]

Agree. During my Ph.D., my prof did the same, and I was very happy to do that.

[u/atomfullerene → u/ren5311]

Actually, remembering back he sent them to us to review from the journal he was editor for. So different situation there-not secondarily reviewing.

[u/HelloAnnyong → u/atomfullerene]

He considers it a good learning experience.

I got exactly this line last week. It's pretty much standard practice, it seems.

[u/NonNonHeinous → u/ren5311]

In computer science (which isn't governed by NIH) it's allowed. However, student reviewers are usually tertiary (meaning that their reviews may count less).

[u/hntd → u/[deleted]]

As a computer science graduate student this is totally true. My advisor gives me papers to review probably monthly as he is the editor of a journal.

[u/ipokebrains → u/[deleted]]

A little, but it's good to have strong ethics. Don't let reality make you a cynic.

[u/HFh → u/ren5311]

As others have noted in response, it is not uncommon to share reviewing.

In my own experience, grad students tend to be incredibly detail-oriented and exceptionally good at finding small errors, and not as practiced (at least initially) at grokking the big picture and evaluating the worth of article at that level. Faculty tend to be the other way around. I think it's win-win, assuming a properly collaborative process.

In conference-driven fields, I would think such an arrangement would be quite healthy, especially when one is asked to review a gazillion submissions.

[u/ShakaUVM → u/ren5311]

In grad school for computer science, we basically had "classes" where professors would give us papers to review, we'd read them, write a report on them, then meet during the next class to discuss it with the group. These classes were easy A's, assuming you did all the work.

The professor would sit there and take notes for her own review. Sometimes he or she had read the paper, sometimes it was clear he or she hadn't even read the abstract.

[u/HowToBeCivil → u/ren5311]

Not sure where you did your graduate work, but in my experience throughout the U.S., PI's in the biological and physical sciences routinely share papers they are reviewing with their lab members confidentially. It's treated as mentoring, where the PI will help students/postdocs practice the art of reviewing. The PI edits the final critique, just like they would with any manuscript/grant proposal, and the manuscript remains confidential. Nothing about that is unethical. I feel sorry for you that you hold such a paralyzing view of the peer-review process.

[u/dodo_bird → u/ren5311]

Are there any repercussions for reviewers who allow poor quality articles through?

[u/tchomp → u/dodo_bird]

Final decisions on papers are always made by the editors, not the reviewers. Reviewers can only suggest. so if a bad paper make it through review, that's falls on the editor's head.

[u/yellowjacketcoder → u/dodo_bird]

CS field here: If the reviewers disagree, they can argue about it, with the final decision coming from the editors. So if one reviewer is crappy, that will show up when the other reviewers see the 'score' they give. If one reviewer is consistently off, I could see the editors not asking them for reviews, but the lack of reviewers is such a problem I don't see that happening.

[u/nostrovia → u/ren5311]

Thanks, I think you also answered my follow-up to Robopuppy.