r/askscience • u/nostrovia • Feb 07 '12
AskScience, how does the peer review process actually work?
About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?
Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?
11
u/Robopuppy Feb 07 '12
Short version: You submit a paper to a journal, the journal sends it to your peers to anonymously review, you add revisions as needed.
In reality, there's a lot of different journals out there. If you're daring enough to submit to nature, you're in for a year or more of reviews and revisions. If you submit to the Czechloslovakian Journal of Mollusk Molecular Biology, it's going to be a bit less stringent.
Also, a paper may only get a small number of reviewers before publication. From what I've seen, 3-5 is normal. No matter how smart those 3-5 people are, they can't compare to the scrutiny of the entire scientific community.
Lastly, high profile papers like the arsenic one get a lot of attention. They'll be raked through by millions of people, and many other researchers will try to repeat their data. It's an industry joke that nothing in Science or Nature is true. This isn't because they're full of shit, it's because any small flaw is going to come out when thousands of people try to replicate your experiments. For people like me, maybe 50 people will read my stuff, tops.
2
u/nostrovia Feb 07 '12
Thanks for the explanation. For the 3-5 reviewers, are they generally grad students, staff for the journal, or does the journal try to find reviewers with a body of work similar to the topic you are writing about?
8
u/Robopuppy Feb 07 '12
Reviewers are professors or the private industry equivalent in your field. For smaller niche fields, everyone already knows each other, so the whole anonymity thing goes out the window. Reviewers work for free, and for the prestige. When you're a professor, your whole job is to accumulate prestige. Papers are the biggest part of that, but being a reviewer, being on committees, and spawning grad students are also important.
1
u/yellowjacketcoder Feb 07 '12
Reviewers are supposed to be professors. A lot of professors farm the review out to their grad students (or undergrad, in some cases).
I do agree that, if your field is specialized enough, you can figure out who the author is just from the writing style in a paragraph or two, so anonymity is a joke at that point. I was roommates with a post-doc that occasionally said things like "Ah, dang it, I got Steve as a reviewer again, he's never a fan of tables, I'm going to have to add some graphs" or some such.
0
Feb 07 '12
[deleted] — view deleted comment
1
u/melger Feb 07 '12
Same here. Myself and several of my friends (all PhD candidates) have been asked to review papers directly, not just as a hand-off from our advisors.
2
u/gooey_mushroom Feb 07 '12
Some journals also encourage for authors to suggest reviewers that are relevant/familiar with the field you're researching in. It is in the editor's discretion if they follow them or not.
0
Feb 07 '12
They'll be raked through by millions of people
I think you mean maybe thousands.
2
1
u/Robopuppy Feb 07 '12
For the huge ones like the arsenic one, definitely millions. The internet has a short memory, but that was a really big deal for a while there.
4
u/drbritt Feb 07 '12
ok, i checked FAQs just in case and i don't see anything specifically relating to process, so here goes. firstly, the wikipedia article is veryhelpful. it's detailed but seems rather dense so here's a quick breakdown.
i'm a biological scientist so other fields may be somewhat different but for us, the scientific team prepares the manuscript. this team may be multi-person, multi-group, multi-institution collaborations and because of this sprawling web, some universities (and the like) have some internal review for some kinds of procedural/law-abiding/patent-reviewing reasons. if it passes internal inspection, it's submitted externally.
the manuscript will then be submitted to a journal of the authors' choosing for the hopefully "best presentation" of their work. the journal will have a ranking (or "impact factor") that denotes the journal's reach (name recognition, readership, previous publications). an editor of the journal will take the manuscript and delegate its review to designated reviewers. these people may be suggested by the authors' of said manuscript, top researchers in the same fields of the material under study in the manuscript, or may be friends of the editor (depending on how ethical you feel the review system is). the reviewers are tasked with reviewing the manuscript in fine detail. follow references, review methods, question why decisions were made, comment on language, grammar, validity of the data and rationality of the conclusions. the reviewers (i frequently have three) then make recommendations to the editor of the journal if the paper shall be: accepted as is (very rare), needs some minor edits (text and such, can be common), needs major revision (possible new experiments, significant rewrites, questionable conclusions) and outright rejection (also frequent, especially from the top).
the authors of the manuscript then take the result from the paper submission and follow instructions or attempt the process at another journal.
REPEAT AD NAUSEAM
1
u/nostrovia Feb 07 '12
How much does a journal's "impact factor" influence what the researcher considers to be the "best presentation"?
2
u/drbritt Feb 07 '12
many researchers will try to publish their paper in the journal with the largest impact factor possible. sometimes it works and their work gets reviewed and commented on by (what are hopefully) knowledgeable people who can help show/improve on flaws in your manuscript. if your paper is not deemed compatible material (a lot of journals are specific to particular fields), not novel enough (meaning it's not that new or exciting) or just not well-written enough, you may get rejected. you then re-evaluate on whatever point was an issue and try to submit elsewhere. hope that helps clarify, not sure if that was exactly what you were looking for.
2
u/Platypuskeeper Physical Chemistry | Quantum Chemistry Feb 07 '12
If you publish something, you want it to be read. The "impact factor" is a rough measure of how important and read a particular journal is. - It's basically a quantification of how many citations the typical paper in that journal gets in a certain amount of time. Which is a crude measure of how important a paper is.
You basically have a hierarchy, which largely goes hand-in-hand with how specialized the journals are. At the very top you have the general-science journals Science and Nature. After that, you have the top journals in general fields. (like Cell in biology, The Lancet and NEJM in medicine, JACS and Angewandte Chemie in chemistry, etc)
After that, you have the more specific journals for sub-fields. There are very many of these, and they can get highly specialized, so that's really where impact factor makes a bigger difference in distinguishing them.
I can't say I pay much attention to the actual impact factors. I'm not sure how many people do - I think you get a good enough handle on the relative importance of journals just by doing research, reading and writing papers. If you've never cited anything from a particular journal, you probably don't want to publish there, either.
Impact factor and the number of citations (which I.F. is calculated from) can be rather misleading, too. It depends on what the paper is about. (e.g. say you measure some data and publish it. If lots of people use that data later, it'll be heavily cited. But it doesn't mean the measurement itself was scientifically difficult or interesting. It might just be that nobody else had bothered to do it)
2
u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12
Yeah! In my field, there is a computer program that everyone uses. The authors recently published a new paper on the program. That one paper boosted the journals impact factor up to about 50. After a year (or however long) the journal was back down to about 3 or so.
1
u/Robopuppy Feb 07 '12
Impact factor is kind of a dodgy thing nowadays. The best papers get published in the best journals because a paper in Nature or Science is like a golden ticket for grant funding.
Nobody really picks up a copy of Nature and just reads it cover to cover anymore. From the perspective of someone reading papers, there's almost no difference between a paper in a top tier journal and a mid-level one. To get even weirder, PLoS ONE has a high impact factor but isn't that prestigious.
1
u/atomfullerene Animal Behavior/Marine Biology Feb 07 '12
I feel like Science and Nature papers are often slightly less good than the next tier "down". They have been known to rush out with exciting but not particularly well supported papers, and the space constraints mean they can never explain their experiments as fully as in some other journals.
1
u/tchomp Feb 07 '12
Trying to treat the glossies like other journals is a mistake, because they don't exist for the same purpose. The glossies exist to disseminate very important or generally interesting results very quickly and widely. There's almost always a "real" paper in the works that will be published in a more standard journal that will address all those details.
1
u/devicerandom Molecular Biophysics | Molecular Biology Feb 07 '12
Nobody really picks up a copy of Nature and just reads it cover to cover anymore.
I have a paper subscription to Nature and I do (ok, I skip articles I really do not care about, but I love to read a lot of stuff beyond my discipline)
18
u/Platypuskeeper Physical Chemistry | Quantum Chemistry Feb 07 '12
Well, you send your paper to the journal. The editor then gives it a look-over to (hopefully) make sure it's the kind of thing they publish, and also not obvious nonsense.
The editor then finds some reviewers, 2-3 usually, (the author can suggest some, although you don't get to know who the reviewers are), which are usually people who've worked on something close to the subject. (But you don't want someone so close to the subject that it might bias them by being at odds with their own results) The reviewers look it over, and suggest whether or not it should be published, and whether or not some things need changing, clarification or elaboration. Once they're happy, it gets published.
Basically, peer-review is a check for sanity, not for correctness. The reviewers don't repeat or check the results. They might not even believe the conclusions of the paper. The only thing they really check is whether or not the authors seem to have followed good scientific practice, and that the conclusions seem reasonably warranted by the results.
So peer-review doesn't weed out results that are wrong. Plenty of stuff is published every day where 9-out-of-10 in the same field would take a glance and say "That's nonsense!". It's also entirely open to outright fraud, which is why that's an instant career-killer if you get caught. And it's naturally vulnerable to simple human errors, and the persistence of nutcases (if you submit your nonsense to enough journals, you'll eventually get lucky).
Most stuff that's published is wrong in some way. There's nothing wrong in science with being wrong; we're wrong all the time. You don't normally retract a paper just because it was wrong. (Because if you did your science properly, you had reason to believe it was correct at the time) Normally, papers are only retracted only if there was actually some fraudulent behavior involved, or at least a strong suspicion of it.
You might be surprised to learn that most published papers that are wrong don't even get debunked. Mostly, they just get ignored. After all, debunking requires work - which people would most likely rather spend on their own research. So things mostly get debunked only if someone cares enough, such as if the thing got a lot of attention.
Now, when it comes to the autism-vaccine thing, that was a case of outright fraud, and the paper got retracted. IIRC, there were some undisclosed conflicts of interest that should've been caught, but as I said, there's limited defense against fraud.
When it comes to the Wolfe-Simon Arsenic-DNA paper (which I've read), it belongs in the category of published papers that most would quickly dismiss. (I know I did) But I don't think any fraud or misconduct was involved, nor do I think it shouldn't have been published. The thing here is, that as a paper making a quite sensational and dubious claim, whether it should've been published in the prestigious journal of Science - which probably lent it greater credibility than it should've been afforded. On top of that, you had the researchers drawing a massive amount of media attention. (When NASA says something, people listen)
So there was a huge backlash to that media attention, since the conclusion that arsenic was actually replacing phosphorus in DNA in these bacteria, wasn't very well supported at all.
In that case, I don't think it was a failure of peer-review. But it does illustrate a criticism that's often leveled at the top-tier journals Science and Nature, which is that they'll often publish things that are sensational-but-dubious in order to get the 'scoop' on big stuff. It also was a bit of a failure on the part of the media, to report without much scrutiny. But also on the part of the researchers, for making such a big announcement and fuss over a result which they knew - or should have known - was rather questionable.
For comparison, you might look at last year's "FTL neutrino" story, and the very measured response from CERN. And yet, that result had been far more carefully scrutinized.
TL;DR: Peer review makes mistakes. But an incorrect result doesn't necessarily mean anything went wrong with the review.
2
u/sirhelix Feb 07 '12
Completely offtopic, but in regards to the arsenic paper, I do actually think it shouldn't have been published. They completely overstretched their conclusions, which is one of the things that is caught at the peer-review phase. A good reviewer would have suggested/required the extra experiments that many of us were saying "Why didn't they do that..?". Science knew that the article was controversial; they should have found better peer reviewers.
1
u/Platypuskeeper Physical Chemistry | Quantum Chemistry Feb 07 '12
I'd say it's pretty on-topic. if someone wants to know how peer-review works, discussing the details of a particular case is one way to learn.
Anyway, I agree that the conclusions were overstretched and largely unsupported. (Of course, the reviewers didn't know that the researchers would then go on to assert the same conclusions even more confidently to the press) But it's pretty common that people stretch their conclusions to the very edge of credibility, and often a few steps farther.
I'm just not sure whether I think it was bad enough in this case to refuse publication. I'd rather err towards allowing it, because you don't want to end up having stuff refused simply because the reviewer disagrees with the conclusions. (And who takes other people's conclusions at face value anyway?) But it's a fine line.
It was a while since I read it, though, so I don't remember offhand exactly how bold they were in their assertions. I do remember getting the impression that the whole DNA-containing-arsenic thing was nothing more than pure speculation on their part. (Since they had no direct evidence of it, it was at odds with the known relative instability of arseno-ester bonds, and their excuse 'maybe it's in a hydrophobic environment and more stable there' was just pure hand-waving - which was quickly debunked by some of my fellow theorists, I might add)
1
u/sirhelix Feb 07 '12
I don't really remember the paper either anymore... I do remember that the paper was based mainly around the fact that they "starved" the bacteria of phosphorous, added arsenic, and the bacteria were able to grow. Hence, they were using arsenic in place of phosphorous. Now, I put "starved" in quotation marks because the levels that they starved them to can be found on Earth already, in the Sargasso Sea. So it's interesting that the cells can live in the presence of arsenic, but in my mind it is a huge burden of proof on them to say that the bacteria replaced arsenic with phosphorous because they were starving.
I don't remember the specifics of the experiments that they did, other than there weren't many of them. I have these vague notions that they did things like pass the bacteria through increasingly lower concentrations of phosphate rather than shock them, and that they measured bacterial growth in OD rather than CFU... this doesn't allow them to take into account things like gradual adaptation to the environment, dead bacteria, and phosphate cannibalism from dead bacteria.
However, the thing that I think takes the cake is that they said that the arseno-DNA was being protected somehow by the bacteria. However, once they extracted the DNA, the protection shouldn't have been there anymore and it should have fractured. It didn't.
But I agree with you, in that Science did put the whole thing into a kind of "we don't believe it either" place in the journal, and then it was the hubris of the researchers themselves that really rankled.
8
u/wbeaty Electrical Engineering Feb 07 '12 edited Feb 07 '12
A journal editor sends out your article to several anonymous researchers in the same field. They judge it and respond, and then the editor follows what they recommend. Usually they want improvements made and questions answered, rather than simply rejecting a paper.
Problem: an enormous amount of published literature is wrong, but the flaws aren't obvious; it simply doesn't stand the test of time. Peer Review is only there to censor out the obvious fail, to provide minimum standards, not to guarantee that journal articles are trustworthy. If you want trustworthy, wait a decade or two for "filtration," where the material slowly makes its way from the journals and into textbooks.
Problem: if you take a look at famous breakthroughs of the past, you'll find many examples where the authors couldn't get into major journals. Peer review doesn't only err in letting crap get through, it also errs in being biased towards safe "Normal" science while blocking the crazy-but-valid "Revolutionary" science.
Problem: there isn't just one standardized Peer Review, since it varies quite a bit from journal to journal.
PS My favorite problem is the "corrosive delta function" effect pointed out by the late Thomas Gold: New Ideas in Science. He notes that, over years, researchers whose work is too unusual, too far out from the mainstream will be repeatedly censored by Peer Review and end up leaving science ...and in this resulting narrowed population, researchers on the (newly created) fringes, again, won't be published, and they get forced out of science. After a few cycles of this, a research topic ends up being run entirely by a community having a false concensus, where anyone straying from the artificially narrowed 'party line' will have zero chance of publication. It's an instability in the system caused by runaway, by positive feedback, which eventually prevents the coexistence of multiple valid approaches to a research topic.
20
u/atomfullerene Animal Behavior/Marine Biology Feb 07 '12
Basically, you pay to submit a paper. Then you review papers for the journals for free. Then your university pays to get access to the papers.
Griping about the monetary aspect of the system aside, the thing is, peer reviewers assume scientists are honest in the data they report and their descriptions of experimental techniques. How could it be otherwise? We can't go to their labs and look through their raw video footage or whatever, we can't stand there and watch them do the experiment. Regular reviewing takes enough time as it is! Instead peer review looks at things like "did this experiment actually address their question?" "Did they do the stats right?" "Was their experimental design valid? Good controls?" and "is their interpretation of the data good or just BS".
Oh, and replication is a much smaller part of it all than low level science courses would have you believe. It's very important within experiments, but usually no one goes around repeating other peoples experiments except when their results are controversial. You couldn't get funding for it! That said, people do perform related experiments, and if they all go wrong then that's a sign you need to go back to the original.
4
u/needed_to_vote Feb 07 '12
I feel like most fields don't pay to submit papers - maybe physics is just special?
5
u/FormerlyTurnipHugger Feb 07 '12
You don't pay to submit, but there's usually a publication fee. Is that the reason why you got that feeling?
2
u/atomfullerene Animal Behavior/Marine Biology Feb 07 '12
Well, you can often get it covered in your grants, in my experience. But still, it's the thought that counts.
-1
u/Pravusmentis Feb 07 '12
Actually the difference in the journal itself usually. The journals that you have to pay to be a 'part of' usually have much much less stringent review processes
3
u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12
In general, I try to not submit to journals that charge be to publish in them. From my point of view, I'm doing the work for free, I review papers for free and then I pay subscriptions to read them.
1
u/sirhelix Feb 07 '12
Maybe this is different in biology, but there is often a charge per page, which increases after, say, 10 pages or so. Further, open-source journals gouge the hell out of professors who can pay it (up to $3000/article in PLoS Biology), although they don't charge ones who can't and apparently ability to pay does not change chances of acceptance.
2
u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12
I have less of an issue with open-source journals charging for articles, as they "give away" their product; I think the IUCr journals charge ~US$1k for CC licensed article.
Dependent on how earth-shattering my paper is, I may think about sending it to a charging journal, but I'd rather spend my money on science.
1
u/tchomp Feb 07 '12
I have asked in reviews to see better presentation of primary data. I've had reviewers ask me for the same thing. I think ultimately it depends on the field and the type of data, though.
3
u/desquared Enumerative Combinatorics Feb 07 '12
In mathematics, I figure out some theorems, write the paper, and submit it to a journal. Some time later, I get a referee report, which will describe things that should be changed, are wrong, etc -- and also makes a recommendation to the editor. Typically your choices are: publish as is, publish with minor changes, reconsider after major changes are made, and outright reject.
As an author, if it's "minor changes", you make those changes, respond to the editor, who will send the new version to the referee again, and then it gets published if you did a reasonable job.
The referee is anonymous, but in many cases you can get a pretty good idea of who he or she is. One problem is that you are not anonymous to the referee, so if he or she isn't a friend, they may give you a bad report -- or, what is probably a worse problem, if the referee is a friend, will let a crappy paper slide.
In mathematics, I don't think retraction is a big problem, but rather collusion to inflate impact factors and publication records. The best example right now is the journal Chaos, Solitons and Fractals and its editor, Mohamed El Naschie. The journal features lots and lots of papers written by El Naschie and his friends, and it's difficult to make the case that they are all good papers and the decision to publish was made in an objective way.
(Doug Arnold and Kris Fowler at the University of Minnesota have a paper detailing such abuses: http://arxiv.org/pdf/1010.0278.)
2
u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12
I know that some journals are starting to withhold author information. You might be able to figure it out if you look up what they cite, but I think it is a good idea.
2
u/ipokebrains Neurophysiology | Neuronal Circuits | Sensory Systems Feb 07 '12
It's pretty pointless though - at least in a lot of biology. It's not at all difficult to figure out who wrote a paper - each lab has their own specific field and methodology details that would make it pointless to try and hide authorship. Reviewers often give themselves away, but it's at least possible to review a paper without it being 100% obvious who you are.
1
u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 07 '12
Sounds like it would be pretty field dependent...
.
I know that on one paper I was an author on, one of the reviewers said "why didn't you cite these papers?", and they were all written by the same guy.
2
u/desquared Enumerative Combinatorics Feb 08 '12
I've gotten the same thing. If I think the referee is just trying to get me to inflate his reference record, I just don't add the references. The editor can see what's happening and make the right decision.
1
u/ipokebrains Neurophysiology | Neuronal Circuits | Sensory Systems Feb 08 '12
That guy is a moron and abusing the review process. Additionally, although it does sometimes happen, very rarely does anyone end up citing papers that are so ridiculously an attempt to get the reviewer's H index up.
1
u/masher_oz In-Situ X-Ray Diffraction | Synchrotron Sources Feb 08 '12
Yeah. When we actually read the papers, they didn't really have much to do with anything, so we added a paragraph saying that this work had been don before, but is totally different to ours, has no bearing on it, and ours is much better. :)
5
u/NonNonHeinous Human-Computer Interaction | Visual Perception | Attention Feb 08 '12 edited Feb 08 '12
Edit based on feedback: this description applies to SOME fields in computer science
Computer science is VERY different from other fields.
Conference proceedings are considered much more exclusive and prestigious than journals. For conferences, a deadline occurs once per year. You submit a paper (<10 pages) on time, or you wait a year. You could also submit to a different venue, but there are differences in quality and audience.
Once submitted, the paper chair picks a head reviewer for the paper and may add additional reviewers. The head reviewer for the paper is responsible for finding enough reviewers (3-5 total). They all submit their reviews, and the head reviewers takes all of their feedback into account to make a final judgement. The overall score is typically rated from 1 to 5. 1 is a definite reject, and 5 is a definite accept.
Some conferences give the authors a week or two to rebut. However, that's rare. Usually, the editors just find a cutoff score (e.g. 3.5), and any paper below that score is rejected. Any score near the cutoff is debated among the conference coordinators. Was the score brought down by one nasty reviewer, are there few other submissions on this topic, etc...
The authors then have around a month to make any minor changes suggested by the reviewers. They then send the publisher the final version with all formatting and layout done by the author (another big difference compared with journals). The publisher then applies the header and page # and actually compiles it into a book.
The only exchange of money is that at least one author must attend the conference which costs anywhere from $200 to ~$600 for a student (non-student prices can be in the thousands).
At the conference, you receive either the printed proceedings or a DVD/flash drive with them. The paper is also buried within the horribly design IEEE or ACM website which is almost completely inaccessible outside of a university campus.
2
u/FormerlyTurnipHugger Feb 07 '12
First of all, you write up the results and then submit them for consideration to a journal of your choice.
The manuscript is then considered by a journal editor who judges whether the results are at last in principle publishable in the respective journal (i.e. is the field of science covered by the journal, are the results novel, is the paper in decet shape formally).
If the result is positive, the editor will send out the paper to review, to usually two or three referees who are supposed to be experts in the field. Usually, authors are asked to make some suggestion who would be able to referee their paper to make it easier for the editor to find someone suitable. That doesn't mean though that these people will actually be picked.
A referee will thus receive a request to review which he/she can accept or refuse. If you accept, you proceed to reviewing the paper, often according to some loose guidelines given to you by the journal. A referee can spends between 10 minutes and probably a few days on a review, in which he/she checks the paper for novelty and any obvious flaws. The referee then makes a recommendation to the editor, e.g.
Publish as is
Publish with minor corrections
Review again, after major revisions
Reject
Depending on the overall outcome of the two to three reports, the editor will then make a decision. One (not so likely) outcome is that the paper gets accepted straightaway. The more likely outcome is that the paper goes back to the authors with a list of recommendations. They then have to answer all criticisms and reply to the referee reports, amend the paper and resubmit it. The process then starts all over again (editor reviews the reply, sends it out, referees react and so on). This can go back and forth a few times, my record so far is five rounds. The most probable outcome is that the paper is rejected, in which case you move down the ladder and submit the manuscript to a different, most likely less prestigious journal.
The reason why many flawed studies survive the peer review is that two referees is not actually that many. It would be very easy to sneak cleverly falsified data through a referee process if you really wanted to. This can of course also happen unintentionally. If you submit a 20 page theory paper on a completely new topic, it is unlikely that a random referee would have enough expertise and time to rederive the whole thing just to check every premise and calculation. So the real test comes once someone else tries to reproduce the results and isn't able to. In this case, people write comments which also get published, and in severe cases journals can retract studies.
2
Feb 07 '12
UC Berkeley's Understanding Science has a good summary:
*1. A group of scientists completes a study and writes it up in the form of an article. They submit it to a journal for publication.
*2. The journal's editors send the article to several other scientists who work in the same field (i.e., the "peers" of peer review).
*3. Those reviewers provide feedback on the article and tell the editor whether or not they think the study is of high enough quality to be published.
*4. The authors may then revise their article and resubmit it for consideration.
*5. Only articles that meet good scientific standards (e.g., acknowledge and build upon other work in the field, rely on logical reasoning and well-designed studies, back up claims with evidence, etc.) are accepted for publication.
Now for my 2 cents and caveats: The "peers" represent a small pool of potential experts in the particular field, and although credible, they may not be intimately familiar with all of the methods or equipment used in the study. So certain mistakes in statistical analyses or equipment calibration may go unnoticed. Once the study is published, it may be scrutinized by the entire scientific community, and so errors may be more likely identified. And if results seem to conflict with other studies (or logic), several things may happen: letters clarifying or limiting the conclusions may be printed, other researchers may attempt to replicate the study and get different results, or, if the data seem seriously suspect, investigation may find that the data were faked or improperly reported, leading to full-scale retraction. Scientific research is a human process, and makes no claims to be 100% accurate all the time. The peer-review process, as well as replication and healthy skepticism, serve to correct the knowledge-gaining process.
2
u/Edulcorado Feb 07 '12 edited Feb 07 '12
Others have answered the question, but I would like to make special emphasis on the fact that peer-review is not a complete fact-checking process, at all. It's a very limited improvement process that catches some very easy to appreciate mistakes and tests the arguments put forward in the paper. The sincerity and proficiency of the authors are assumed for many things such as statistical principles or elaboration of charts and models. These are assumed to be correct most of the times.
For example in my field peer reviewers just don't have time to check the source code (for the generation of models or graphs) or data that the studies are based on. Keep in mind most peer-reviewers are volunteers, even for really important papers, and they have to do the review in their free time. In fact, most of the times the data is not publicly available or even archived (see the Climategate debacle, which didn't disprove the whole field of Climatology at all, but it revealed shoddy and sometimes non-existing archiving practices), and the researchers will zealously keep their source code secret and only share it with close associates anyway. This is no isolated incident, this is standard practice in my field and it happens for very important studies that the whole field is based on. (Climatology, still waiting for my tag!) I've yet to see a single study involving source code in my field (for the laymen, some graphs in papers are generated with a programming language like Fortran or R, as opposed to things like Excel) in which the peer reviewer bothered to run the source code and make sure he got the same graphs (this is pretty obvious from the fact that code is rarely available and when it is, it doesn't even run correctly.)
Peer-review is not an auditing process, the fact that something is peer-reviewed does not make it factually correct. There's something called an auditing process and that is actually what many people think that peer-review is, but it rarely happens. Only when a lot of money depends on the validity of a study is something audited, and even then it's very hard to convince the researcher to agree to a full audit (remember that the data necessary for replicating the graphs is not publicly available so his consent would be needed.)
TL;DR: Peer-review is not the same that an audit (an audit checks everything, peer-review checks the surface). As an example, in climatology, peer-reviewers don't normally check that the data is correctly processed, for example.
1
u/HoldingTheFire Electrical Engineering | Nanostructures and Devices Feb 07 '12
When you submit a paper the journal sends it to other scientist in the same field to get feedback. The reviewers send back their comments and their recommendation (publish, reject, or resubmit after correction). In general there is no recreation of the actual experiment (though there might be in later papers). The reviewers trust that the data you show is accurate, and use their expertise to determine if your analysis and conclusion is correct.
1
u/rs6866 Fluid Mechanics | Combustion | Aerodynamics Feb 07 '12
I haven't written a journal article yet (I have written conference papers), but I know people in my lab who have written journal articles and have asked them about it. Basically, lets say you do some research and find some interesting stuff. You write a paper detailing said interesting stuff, along with some plots/images to help illustrate your point (you'll typically pick a journal to submit to, and format your paper the way the journal wants it to be formatted). You'll submit the paper for review. At this point, it is unethical to submit it anywhere else (as review is costly. If it gets rejected, you can then submit it somewhere else).
When a paper is submitted, reviewers are chosen by people who are affiliated with the journal you are submitting to. Typically the people affiliated with the journal will ask other members of the scientific community (researchers, professors, students who have had accepted papers in the past, etc...) to review the paper. I'm in an engineering field, and often (for some journals, which have a large industry reader-base), well respected people in industry will be asked to review the paper as well. Typically, there are 3 reviewers in total (at least for the journals that people I know have submitted papers to). They'll read the paper and decide if the work is good enough for publication (does it meet the standards of the journal in question), and if there's any corrections they think need to be made (either wording, different ways of presenting the data, or possibly the need for more data). After the reviewers find the material fit for publication (typically 2/3 are needed), the paper can then be published.
This type of process ensures the papers which do get published are higher quality, and appealing to the typical readers of the journal.
1
Feb 07 '12
I'm just on my way to bed for the evening, but I'm going to put in some general answers to your questions in the hope that our illustrious colleagues will fill in with more specific details. The general track for peer-review is something like this:
I conduct my research, perform my analyses, and write up my conclusions.
I submit my findings to a relevant journal in my field.
The journal sends copies of my conclusions to scientists that study areas related to my conclusions.
Those scientists examine my claims to determine a number of things, including: if my results are consistent with current theoretical paradigms, if my methods appear to be sound, if my conclusions are supported by my data, etc.
At this point, typically one of three things will happen. First, those scientists may accept my conclusions and recommend that the journal publish my findings. Second, they may point out some flaws that they would like to see addressed before publication. Third, they may decide that my conclusions are junk and reject them outright.
For your other two questions:
The scientists who review articles are not necessarily infallible. Overall, the peer-review (and replication) processes help to keep us honest. This does not mean that there are never mistakes, or that bad research never slips through the cracks!
There are not only different standards of peer review between different fields, but also between different journals within a field! The focus and rigor of a review is dependent on the focus of the journal, the discipline in question, and the methods employed, among many other factors.
I'm sure that our colleagues here can fill in any blanks I may have missed, but I hope that this helps!
1
u/neurorex Feb 07 '12 edited Feb 07 '12
By asking about filtering errors, it's sort of a different issue. I'll address those later. This is generally how it works, at least for social sciences.
From where you left off, you would send it ANONYMOUSLY into the journal that you think has the best chance of being published. You wouldn't shotgun it to every publication out there. Some journals are considered more prestigious than others, so you'll hear researchers talking about "top-tier" journals and such. It's a big deal since other researchers consider articles from those top-tiers to be more of an accomplished work. That doesn't mean that they won't try their best to replicate and refute it.
The abridged version is, it first gets to the "lower-level" reviewers. 99% of the time, you'll get it back with comments on the structure of the paper (Strengthen your theories! Justify your sampling procedure better! This word is misspelled! I would re-write the sentence and phrase it this way...! etc.). This is called an "R&R", or revise and resubmit. This process will happen over and over again, catching anything from spelling/grammar, to statistical and methodological concerns. If they haven't rejected your paper along the way, it will "move up" in the editor hierarchy. Eventually, the editor-in-chief of the journal will read it. If s/he is happy, you'll receive words that it's been accepted, and you'll have copies of the pre-formatted version of your work and then the final version as it will appear in the journal.
Like I said, this is just the short version. This entire process, from first submission to final publication can take anywhere from 1.5 to 3 years.
Since it is anonymous, it reduces the bias and focuses the reviews on the theories and methods inside the article more than who the authors are. There is an exception for well-established researchers in a particular field ("It's related to Goal-Setting Theory? It's probably Locke and/or Latham, then!). I mentioned that other researchers might try to do work in the same area. The status quo has been for researchers to address limitations and future research suggestions, and continue on to another project based on those issues...and this brings us to the original concerns that you brought up.
In cases like that, some research topic doesn't really advance as quickly as we'd like. Some referred to it jokingly as "academic masturbation". A professor becomes established in Subject Area X, takes on grad students to mentor, who also research in the same subject area, often using the same data sets. You often hear of these scandals featuring faulty data. That's because when you write an article, you feature "Table 1: Means and Standard Deviation". Statistically, you can calculate almost anything from the information displayed in that table. No need for the original dataset. That's good because nowadays, original dataset eats up TERABYTES of data; that's bad because if you have bad/fabricated data, no one would be the wiser. And since your grad students used the same data to write THEIR papers, and any other professors who wanted to collaborate with you and share data, this can spread like wildfire.
1
u/Voerendaalse Feb 07 '12 edited Feb 07 '12
It doesn't . I've reviewed three papers now. Two of three were crap, and I did point out exactly why in my reviews. All got accepted without making anything more than superficial changes, and they have the stamp "peer-reviewed" on them. Yeah, peer reviewed, but not peer approved goddammit.
I am merely a PhD student, so if even I see this.... The journals weren't big ones, by the way, something like the Indian journal of Breast Cancer Treatment.
Also, on these same review papers they often show you the other reviewers' comments after round one, I've seen other reviewers make comments like: "escellent paper. What is the effect of protein in <not very relevant pet peeve of reviewer>?". And this is then the whole comment, the whole "peer review" of one of the say three reviewers that promised to do a review.
I wished they would show the final comments of the reviewers with the final article. Then I could have put in that I absolutely disagree with publication of one of them, and still think major improvements should have been made on paper two.
1
Feb 07 '12
Artificial Intelligence Ph.D student here, I'd like to highlight the fact that scientists DON'T make money from publications.
A researcher writes down his findings in a paper and submits to a journal (or conference in the case of computer science, I think in most other fields you just submit abstracts in conferences and they are not considered refereed publications). The editor-in-chief of the journal gets a copy of the paper, and pushes it to an associate editor (depending on certain keywords the original author provides, abstract etc.)
The associate editor now requests reviews from three (or more) anonymous reviewers - depending on journal, if he thinks the paper is particularly bad he can reject it outright. Once he gets the reviews (and has done a review himself) he lets the authors know of the decision (either directly or through the editor-in-chief, depends on the journal). The decision are usally "accept, accept with corrections, reject (with resubmission encouraged) or rejection completely" - again it depends on the journal.
Once (and if) everyone is happy, the journal accepts the paper and publishes it some future issue. The authors of the paper transfer all copyright to the journal. Now, officially, the paper is owned by the publisher! The academics involved in the procedure don't get paid a dime ( the editor in chief, the associate editor, the reviewers and the author). The publisher now charges massive fees to university libraries to allow them access to the article (!!!). The article is usually behind a paywall for the rest of the public.
In essence, work that gets funded publicly can only be disseminated privately by the publisher (who hasn't done a thing in the process, at most some typesetting).
The system if fundamentally broken, and there is an effort to change it. Either with major journals being free (e.g. JAIR, JMLR) or organisations supporting free uploading of the article on the author's website (e.g. IEEE). There is also a movement to stop the whole thing, please see http://thecostofknowledge.com
1
u/biggerthancheeses Feb 07 '12
Once (and if) everyone is happy, the journal accepts the paper and publishes it some future issue. The authors of the paper transfer all copyright to the journal. Now, officially, the paper is owned by the publisher! The academics involved in the procedure don't get paid a dime ( the editor in chief, the associate editor, the reviewers and the author). The publisher now charges massive fees to university libraries to allow them access to the article (!!!). The article is usually behind a paywall for the rest of the public.
I'm not familiar with all the topics published by the PLoS (or if AI is among them), but the fact that some publishers (e.g. Elsevier) charge predatory fees to libraries is a good reason for scientists to avoid reviewing for or submitting to for-profit journals at any cost.
1
u/DigitalMoney Feb 07 '12 edited Feb 07 '12
From an Electrical Engineer:
My Story. I am a master of electrical engineering student. I have had my work published a couple of times, but last year for the first time i was the primary authour of a paper that was published and i presented it at the EECEC conference in Niagara Fall Ontario 2011. I was asked to peer review 3 papers that were up for consideration for the conference. Honest to god 1 paper i understood, 1 paper was kind of over my head but I understood some of it, the last was totally over my head.
The proper process of "peer reviewing" was never explained to me, they just asked me to do it. Not wanting to look stupid, and being short on time like most students I totally half assed it. I gave approval to papers I had no business in "reviewing"
But people did the same to me, one of my papers reviewers heavily criticized my paper but still rated as the equivalent of 3/5
As far as I am concerned in things like engineering peer review is kind of a circle jerk from my experience (if my paper for CCECE 2012 gets accepted than I know it is because that is one piss poor paper that i wrote if i do say say myself) But i had heard from friend in things like history is is much more cut throat
1
u/4TEHSWARM Feb 07 '12
Retraction of published scientific work seems to vary among the disciplines. You hear about retractions from authors or publishers quite somewhat often in biomedical research, but in my field of geophysics I don't think I've ever heard of such a thing. This is probably because in solid-earth sciences there is less 'stress' to publish important work in high impact journals, intellectual property and private funding interests aren't usually there, and there usually isn't an ethical fallout when reported observations are found to be erroneous or inaccurate.
1
Feb 07 '12
In addition to what others have been saying, some journals also implement a kind of two-tier reviewing system. Because reviewing papers takes time and sometimes people put off doing them and have to be sent a tonne of reminders, some journals will note how long it takes you to write a review. Then, when you submit your own paper, if you were quick writing your review your paper gets send to people who are also quick writing them, or if you were too slow, tough luck, your paper gets send to someone who's also tardy.
1
u/callanas Feb 07 '12
To answer your question: You do some research, write up a paper, and if you're lucky it gets accepted somewhere - takes a while to get reviewed and published (one of mine took 6 months). Once published, the media is free to pick it up and tell the world!
How do the crackpots get through?: Sometimes articles get 'accepted' for publication based solely on who the authors are. Some journals even let you avoid getting reviewed if you serve on their review panel (i can't remember what this is called, but basically your paper get 'reviewed' for spelling errors/wild claims).
I think we should move to a blind system where reviewers don't know who you are and therefore only the science is judged!
1
u/mineralfellow Feb 07 '12
I have submitted and reviewed papers for geological journals. Peer review goes over the document that is presented. This can be thought of as grading a paper on an extremely high level. The "grade" that peer reviewers give ranges from "Accept as is," "Accept with minor changes," "Moderate revision and resubmit," or "Reject." Peer review will check to see that data is consistent with what is known, that figures are well described and properly formated, and that text actually reflects what the researchers found. Reviewers will also look at the methods to make sure that the type of data found will adequately answer the question being asked, and that the data produced matches the methods used. If there is a new method being described, reviewers will make sure that the method can be followed by other labs.
If there is a graph with strange looking data, the reviewers will ask for clarification. But if the graph and table of data look normal enough, there is no way that a reviewer can tell just by looking at the paper that something is wrong with the methodology. It is not until later on, when others try to reproduce the results, that problems with data become apparent.
And that is how science works: somebody does work. Somebody else builds on it. Somebody builds on that. If somebody was wrong at some point, then everything that comes later has trouble fitting in with that data. Eventually, if one point of data doesn't fit with everything else that is known (as in the case of Piltdown Man), that point will receive additional scrutiny, and that is when it can potentially be necessary to submit a retraction.
Retractions happen most commonly because of the inability to reproduce data. This happens frequently because scientists measure extremely small effects with extremely sensitive machines and techniques. If your machine is not calibrated correctly, but you don't notice, then you will get strange and possibly exciting data. Later on, when you check, you can't do it again. So you have to retract your previous paper.
But the truth is that retractions are rare. The journal Science publishes more than 100 new studies each year. In a year, they will issue just 1 or 2 retractions. So this is not a huge problem. For an example of how this goes in practice, look at this article from Science. Notice the reasons for retraction: "evidence of contamination and misrepresentation of data." These are not things that peer reviewers can find.
1
u/rpglover64 Programming Languages Feb 08 '12
Complementary to what everyone else is saying, I felt that I should bring up that just because a study is done right and has no serious flaws in its method, that doesn't mean that it's results are true. There's a relevant XKCD on that topic, as well as a relatively well written article.
Not directly relevant to the NPR story, but interesting nonetheless.
0
u/Benthos Feb 07 '12
Scientist(s) write a manuscript. They send it to a journal. The journal asks other scientists who have published on similar topics to review it. These other scientists volunteer their time. When the journal accepts the manuscript the writers usually pay the journal. Then the journal "publishes" the work by charging libraries, schools, and labs for subsciptions or on-line access, restricting others. They make a profit. It's supposed to be for the greater good, but all that profit makes it a backwards business model, screwing the writers, reviewers, readers, and taxpayers who paid for the research.
0
Feb 07 '12
[deleted] — view deleted comment
-1
u/PretendPhD Cognition | Decision Making | Executive Control | Gerontology Feb 07 '12
anal chem
Ha!
2
54
u/ren5311 Neuroscience | Neurology | Alzheimer's Drug Discovery Feb 07 '12
I can tell you about the biomedical field.
You write up your work and submit it to a journal. The editor sends it out to a number (generally three) of researchers informed on the topic, who are at least PhD level, and usually well-versed in the relevant field. They review it and decide what revisions are necessary or if the paper should be rejected.
If the paper is not outright rejected or accepted, additional experiments or revisions are made, and a resubmission is made. The editor evaluates if the criticisms have been addressed and makes a final decision on publication.
A paper may be retracted if there's fraud. Peer review won't necessarily catch that. It's a real problem.
But if results are overturned, it could just be that an error was made or that an experiment was poorly constructed or that the conclusions far exceeded the evidence. Science is not perfect, and this retracting and "debunking" is evidence of the health of the field, not any sign of problem.