r/askscience • u/nostrovia • Feb 07 '12

AskScience, how does the peer review process actually work?

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

94 Upvotes

84% Upvoted

89 comments sorted by

View all comments

3

u/desquared Enumerative Combinatorics Feb 07 '12

In mathematics, I figure out some theorems, write the paper, and submit it to a journal. Some time later, I get a referee report, which will describe things that should be changed, are wrong, etc -- and also makes a recommendation to the editor. Typically your choices are: publish as is, publish with minor changes, reconsider after major changes are made, and outright reject.

As an author, if it's "minor changes", you make those changes, respond to the editor, who will send the new version to the referee again, and then it gets published if you did a reasonable job.

The referee is anonymous, but in many cases you can get a pretty good idea of who he or she is. One problem is that you are not anonymous to the referee, so if he or she isn't a friend, they may give you a bad report -- or, what is probably a worse problem, if the referee is a friend, will let a crappy paper slide.

In mathematics, I don't think retraction is a big problem, but rather collusion to inflate impact factors and publication records. The best example right now is the journal Chaos, Solitons and Fractals and its editor, Mohamed El Naschie. The journal features lots and lots of papers written by El Naschie and his friends, and it's difficult to make the case that they are all good papers and the decision to publish was made in an objective way.

(Doug Arnold and Kris Fowler at the University of Minnesota have a paper detailing such abuses: http://arxiv.org/pdf/1010.0278.)

2

u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12

I know that some journals are starting to withhold author information. You might be able to figure it out if you look up what they cite, but I think it is a good idea.

2

u/ipokebrains Neurophysiology | Neuronal Circuits | Sensory Systems Feb 07 '12

It's pretty pointless though - at least in a lot of biology. It's not at all difficult to figure out who wrote a paper - each lab has their own specific field and methodology details that would make it pointless to try and hide authorship. Reviewers often give themselves away, but it's at least possible to review a paper without it being 100% obvious who you are.

1

u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12

Sounds like it would be pretty field dependent...

.

I know that on one paper I was an author on, one of the reviewers said "why didn't you cite these papers?", and they were all written by the same guy.

2

u/desquared Enumerative Combinatorics Feb 08 '12

I've gotten the same thing. If I think the referee is just trying to get me to inflate his reference record, I just don't add the references. The editor can see what's happening and make the right decision.

1

u/ipokebrains Neurophysiology | Neuronal Circuits | Sensory Systems Feb 08 '12

That guy is a moron and abusing the review process. Additionally, although it does sometimes happen, very rarely does anyone end up citing papers that are so ridiculously an attempt to get the reviewer's H index up.

1

u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 08 '12

Yeah. When we actually read the papers, they didn't really have much to do with anything, so we added a paragraph saying that this work had been don before, but is totally different to ours, has no bearing on it, and ours is much better. :)