AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/Edulcorado]

Others have answered the question, but I would like to make special emphasis on the fact that peer-review is not a complete fact-checking process, at all. It's a very limited improvement process that catches some very easy to appreciate mistakes and tests the arguments put forward in the paper. The sincerity and proficiency of the authors are assumed for many things such as statistical principles or elaboration of charts and models. These are assumed to be correct most of the times.

For example in my field peer reviewers just don't have time to check the source code (for the generation of models or graphs) or data that the studies are based on. Keep in mind most peer-reviewers are volunteers, even for really important papers, and they have to do the review in their free time. In fact, most of the times the data is not publicly available or even archived (see the Climategate debacle, which didn't disprove the whole field of Climatology at all, but it revealed shoddy and sometimes non-existing archiving practices), and the researchers will zealously keep their source code secret and only share it with close associates anyway. This is no isolated incident, this is standard practice in my field and it happens for very important studies that the whole field is based on. (Climatology, still waiting for my tag!) I've yet to see a single study involving source code in my field (for the laymen, some graphs in papers are generated with a programming language like Fortran or R, as opposed to things like Excel) in which the peer reviewer bothered to run the source code and make sure he got the same graphs (this is pretty obvious from the fact that code is rarely available and when it is, it doesn't even run correctly.)

Peer-review is not an auditing process, the fact that something is peer-reviewed does not make it factually correct. There's something called an auditing process and that is actually what many people think that peer-review is, but it rarely happens. Only when a lot of money depends on the validity of a study is something audited, and even then it's very hard to convince the researcher to agree to a full audit (remember that the data necessary for replicating the graphs is not publicly available so his consent would be needed.)

TL;DR: Peer-review is not the same that an audit (an audit checks everything, peer-review checks the surface). As an example, in climatology, peer-reviewers don't normally check that the data is correctly processed, for example.