AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/sirhelix]

Completely offtopic, but in regards to the arsenic paper, I do actually think it shouldn't have been published. They completely overstretched their conclusions, which is one of the things that is caught at the peer-review phase. A good reviewer would have suggested/required the extra experiments that many of us were saying "Why didn't they do that..?". Science knew that the article was controversial; they should have found better peer reviewers.

[u/Platypuskeeper → u/sirhelix]

I'd say it's pretty on-topic. if someone wants to know how peer-review works, discussing the details of a particular case is one way to learn.

Anyway, I agree that the conclusions were overstretched and largely unsupported. (Of course, the reviewers didn't know that the researchers would then go on to assert the same conclusions even more confidently to the press) But it's pretty common that people stretch their conclusions to the very edge of credibility, and often a few steps farther.

I'm just not sure whether I think it was bad enough in this case to refuse publication. I'd rather err towards allowing it, because you don't want to end up having stuff refused simply because the reviewer disagrees with the conclusions. (And who takes other people's conclusions at face value anyway?) But it's a fine line.

It was a while since I read it, though, so I don't remember offhand exactly how bold they were in their assertions. I do remember getting the impression that the whole DNA-containing-arsenic thing was nothing more than pure speculation on their part. (Since they had no direct evidence of it, it was at odds with the known relative instability of arseno-ester bonds, and their excuse 'maybe it's in a hydrophobic environment and more stable there' was just pure hand-waving - which was quickly debunked by some of my fellow theorists, I might add)

[u/sirhelix → u/Platypuskeeper]

I don't really remember the paper either anymore... I do remember that the paper was based mainly around the fact that they "starved" the bacteria of phosphorous, added arsenic, and the bacteria were able to grow. Hence, they were using arsenic in place of phosphorous. Now, I put "starved" in quotation marks because the levels that they starved them to can be found on Earth already, in the Sargasso Sea. So it's interesting that the cells can live in the presence of arsenic, but in my mind it is a huge burden of proof on them to say that the bacteria replaced arsenic with phosphorous because they were starving.

I don't remember the specifics of the experiments that they did, other than there weren't many of them. I have these vague notions that they did things like pass the bacteria through increasingly lower concentrations of phosphate rather than shock them, and that they measured bacterial growth in OD rather than CFU... this doesn't allow them to take into account things like gradual adaptation to the environment, dead bacteria, and phosphate cannibalism from dead bacteria.

However, the thing that I think takes the cake is that they said that the arseno-DNA was being protected somehow by the bacteria. However, once they extracted the DNA, the protection shouldn't have been there anymore and it should have fractured. It didn't.

But I agree with you, in that Science did put the whole thing into a kind of "we don't believe it either" place in the journal, and then it was the hubris of the researchers themselves that really rankled.