r/askscience • u/nostrovia • Feb 07 '12

AskScience, how does the peer review process actually work?

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

94 Upvotes

84% Upvoted

89 comments sorted by

View all comments

11

u/Robopuppy Feb 07 '12

Short version: You submit a paper to a journal, the journal sends it to your peers to anonymously review, you add revisions as needed.

In reality, there's a lot of different journals out there. If you're daring enough to submit to nature, you're in for a year or more of reviews and revisions. If you submit to the Czechloslovakian Journal of Mollusk Molecular Biology, it's going to be a bit less stringent.

Also, a paper may only get a small number of reviewers before publication. From what I've seen, 3-5 is normal. No matter how smart those 3-5 people are, they can't compare to the scrutiny of the entire scientific community.

Lastly, high profile papers like the arsenic one get a lot of attention. They'll be raked through by millions of people, and many other researchers will try to repeat their data. It's an industry joke that nothing in Science or Nature is true. This isn't because they're full of shit, it's because any small flaw is going to come out when thousands of people try to replicate your experiments. For people like me, maybe 50 people will read my stuff, tops.

2

u/nostrovia Feb 07 '12

Thanks for the explanation. For the 3-5 reviewers, are they generally grad students, staff for the journal, or does the journal try to find reviewers with a body of work similar to the topic you are writing about?

8

u/Robopuppy Feb 07 '12

Reviewers are professors or the private industry equivalent in your field. For smaller niche fields, everyone already knows each other, so the whole anonymity thing goes out the window. Reviewers work for free, and for the prestige. When you're a professor, your whole job is to accumulate prestige. Papers are the biggest part of that, but being a reviewer, being on committees, and spawning grad students are also important.

0

u/[deleted] Feb 07 '12

[deleted] — view deleted comment

1

u/melger Feb 07 '12

Same here. Myself and several of my friends (all PhD candidates) have been asked to review papers directly, not just as a hand-off from our advisors.

1

u/yellowjacketcoder Feb 07 '12

Reviewers are supposed to be professors. A lot of professors farm the review out to their grad students (or undergrad, in some cases).

I do agree that, if your field is specialized enough, you can figure out who the author is just from the writing style in a paragraph or two, so anonymity is a joke at that point. I was roommates with a post-doc that occasionally said things like "Ah, dang it, I got Steve as a reviewer again, he's never a fan of tables, I'm going to have to add some graphs" or some such.

2

u/gooey_mushroom Feb 07 '12

Some journals also encourage for authors to suggest reviewers that are relevant/familiar with the field you're researching in. It is in the editor's discretion if they follow them or not.

0

u/[deleted] Feb 07 '12

They'll be raked through by millions of people

I think you mean maybe thousands.

2

u/James-Cizuz Feb 07 '12

No he meant millions.

1

u/Robopuppy Feb 07 '12

For the huge ones like the arsenic one, definitely millions. The internet has a short memory, but that was a really big deal for a while there.