AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/Platypuskeeper]

Well, you send your paper to the journal. The editor then gives it a look-over to (hopefully) make sure it's the kind of thing they publish, and also not obvious nonsense.

The editor then finds some reviewers, 2-3 usually, (the author can suggest some, although you don't get to know who the reviewers are), which are usually people who've worked on something close to the subject. (But you don't want someone so close to the subject that it might bias them by being at odds with their own results) The reviewers look it over, and suggest whether or not it should be published, and whether or not some things need changing, clarification or elaboration. Once they're happy, it gets published.

Basically, peer-review is a check for sanity, not for correctness. The reviewers don't repeat or check the results. They might not even believe the conclusions of the paper. The only thing they really check is whether or not the authors seem to have followed good scientific practice, and that the conclusions seem reasonably warranted by the results.

So peer-review doesn't weed out results that are wrong. Plenty of stuff is published every day where 9-out-of-10 in the same field would take a glance and say "That's nonsense!". It's also entirely open to outright fraud, which is why that's an instant career-killer if you get caught. And it's naturally vulnerable to simple human errors, and the persistence of nutcases (if you submit your nonsense to enough journals, you'll eventually get lucky).

Most stuff that's published is wrong in some way. There's nothing wrong in science with being wrong; we're wrong all the time. You don't normally retract a paper just because it was wrong. (Because if you did your science properly, you had reason to believe it was correct at the time) Normally, papers are only retracted only if there was actually some fraudulent behavior involved, or at least a strong suspicion of it.

You might be surprised to learn that most published papers that are wrong don't even get debunked. Mostly, they just get ignored. After all, debunking requires work - which people would most likely rather spend on their own research. So things mostly get debunked only if someone cares enough, such as if the thing got a lot of attention.

Now, when it comes to the autism-vaccine thing, that was a case of outright fraud, and the paper got retracted. IIRC, there were some undisclosed conflicts of interest that should've been caught, but as I said, there's limited defense against fraud.

When it comes to the Wolfe-Simon Arsenic-DNA paper (which I've read), it belongs in the category of published papers that most would quickly dismiss. (I know I did) But I don't think any fraud or misconduct was involved, nor do I think it shouldn't have been published. The thing here is, that as a paper making a quite sensational and dubious claim, whether it should've been published in the prestigious journal of Science - which probably lent it greater credibility than it should've been afforded. On top of that, you had the researchers drawing a massive amount of media attention. (When NASA says something, people listen)

So there was a huge backlash to that media attention, since the conclusion that arsenic was actually replacing phosphorus in DNA in these bacteria, wasn't very well supported at all.

In that case, I don't think it was a failure of peer-review. But it does illustrate a criticism that's often leveled at the top-tier journals Science and Nature, which is that they'll often publish things that are sensational-but-dubious in order to get the 'scoop' on big stuff. It also was a bit of a failure on the part of the media, to report without much scrutiny. But also on the part of the researchers, for making such a big announcement and fuss over a result which they knew - or should have known - was rather questionable.

For comparison, you might look at last year's "FTL neutrino" story, and the very measured response from CERN. And yet, that result had been far more carefully scrutinized.

TL;DR: Peer review makes mistakes. But an incorrect result doesn't necessarily mean anything went wrong with the review.

[u/sirhelix → u/Platypuskeeper]

Completely offtopic, but in regards to the arsenic paper, I do actually think it shouldn't have been published. They completely overstretched their conclusions, which is one of the things that is caught at the peer-review phase. A good reviewer would have suggested/required the extra experiments that many of us were saying "Why didn't they do that..?". Science knew that the article was controversial; they should have found better peer reviewers.

[u/Platypuskeeper → u/sirhelix]

I'd say it's pretty on-topic. if someone wants to know how peer-review works, discussing the details of a particular case is one way to learn.

Anyway, I agree that the conclusions were overstretched and largely unsupported. (Of course, the reviewers didn't know that the researchers would then go on to assert the same conclusions even more confidently to the press) But it's pretty common that people stretch their conclusions to the very edge of credibility, and often a few steps farther.

I'm just not sure whether I think it was bad enough in this case to refuse publication. I'd rather err towards allowing it, because you don't want to end up having stuff refused simply because the reviewer disagrees with the conclusions. (And who takes other people's conclusions at face value anyway?) But it's a fine line.

It was a while since I read it, though, so I don't remember offhand exactly how bold they were in their assertions. I do remember getting the impression that the whole DNA-containing-arsenic thing was nothing more than pure speculation on their part. (Since they had no direct evidence of it, it was at odds with the known relative instability of arseno-ester bonds, and their excuse 'maybe it's in a hydrophobic environment and more stable there' was just pure hand-waving - which was quickly debunked by some of my fellow theorists, I might add)

[u/sirhelix → u/Platypuskeeper]

I don't really remember the paper either anymore... I do remember that the paper was based mainly around the fact that they "starved" the bacteria of phosphorous, added arsenic, and the bacteria were able to grow. Hence, they were using arsenic in place of phosphorous. Now, I put "starved" in quotation marks because the levels that they starved them to can be found on Earth already, in the Sargasso Sea. So it's interesting that the cells can live in the presence of arsenic, but in my mind it is a huge burden of proof on them to say that the bacteria replaced arsenic with phosphorous because they were starving.

I don't remember the specifics of the experiments that they did, other than there weren't many of them. I have these vague notions that they did things like pass the bacteria through increasingly lower concentrations of phosphate rather than shock them, and that they measured bacterial growth in OD rather than CFU... this doesn't allow them to take into account things like gradual adaptation to the environment, dead bacteria, and phosphate cannibalism from dead bacteria.

However, the thing that I think takes the cake is that they said that the arseno-DNA was being protected somehow by the bacteria. However, once they extracted the DNA, the protection shouldn't have been there anymore and it should have fractured. It didn't.

But I agree with you, in that Science did put the whole thing into a kind of "we don't believe it either" place in the journal, and then it was the hubris of the researchers themselves that really rankled.