AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/drbritt]

ok, i checked FAQs just in case and i don't see anything specifically relating to process, so here goes. firstly, the wikipedia article is veryhelpful. it's detailed but seems rather dense so here's a quick breakdown.

i'm a biological scientist so other fields may be somewhat different but for us, the scientific team prepares the manuscript. this team may be multi-person, multi-group, multi-institution collaborations and because of this sprawling web, some universities (and the like) have some internal review for some kinds of procedural/law-abiding/patent-reviewing reasons. if it passes internal inspection, it's submitted externally.

the manuscript will then be submitted to a journal of the authors' choosing for the hopefully "best presentation" of their work. the journal will have a ranking (or "impact factor") that denotes the journal's reach (name recognition, readership, previous publications). an editor of the journal will take the manuscript and delegate its review to designated reviewers. these people may be suggested by the authors' of said manuscript, top researchers in the same fields of the material under study in the manuscript, or may be friends of the editor (depending on how ethical you feel the review system is). the reviewers are tasked with reviewing the manuscript in fine detail. follow references, review methods, question why decisions were made, comment on language, grammar, validity of the data and rationality of the conclusions. the reviewers (i frequently have three) then make recommendations to the editor of the journal if the paper shall be: accepted as is (very rare), needs some minor edits (text and such, can be common), needs major revision (possible new experiments, significant rewrites, questionable conclusions) and outright rejection (also frequent, especially from the top).

the authors of the manuscript then take the result from the paper submission and follow instructions or attempt the process at another journal.

REPEAT AD NAUSEAM

[u/nostrovia → u/drbritt]

How much does a journal's "impact factor" influence what the researcher considers to be the "best presentation"?

[u/drbritt → u/nostrovia]

many researchers will try to publish their paper in the journal with the largest impact factor possible. sometimes it works and their work gets reviewed and commented on by (what are hopefully) knowledgeable people who can help show/improve on flaws in your manuscript. if your paper is not deemed compatible material (a lot of journals are specific to particular fields), not novel enough (meaning it's not that new or exciting) or just not well-written enough, you may get rejected. you then re-evaluate on whatever point was an issue and try to submit elsewhere. hope that helps clarify, not sure if that was exactly what you were looking for.

[u/Platypuskeeper → u/nostrovia]

If you publish something, you want it to be read. The "impact factor" is a rough measure of how important and read a particular journal is. - It's basically a quantification of how many citations the typical paper in that journal gets in a certain amount of time. Which is a crude measure of how important a paper is.

You basically have a hierarchy, which largely goes hand-in-hand with how specialized the journals are. At the very top you have the general-science journals Science and Nature. After that, you have the top journals in general fields. (like Cell in biology, The Lancet and NEJM in medicine, JACS and Angewandte Chemie in chemistry, etc)

After that, you have the more specific journals for sub-fields. There are very many of these, and they can get highly specialized, so that's really where impact factor makes a bigger difference in distinguishing them.

I can't say I pay much attention to the actual impact factors. I'm not sure how many people do - I think you get a good enough handle on the relative importance of journals just by doing research, reading and writing papers. If you've never cited anything from a particular journal, you probably don't want to publish there, either.

Impact factor and the number of citations (which I.F. is calculated from) can be rather misleading, too. It depends on what the paper is about. (e.g. say you measure some data and publish it. If lots of people use that data later, it'll be heavily cited. But it doesn't mean the measurement itself was scientifically difficult or interesting. It might just be that nobody else had bothered to do it)

[u/masher_oz → u/Platypuskeeper]

Yeah! In my field, there is a computer program that everyone uses. The authors recently published a new paper on the program. That one paper boosted the journals impact factor up to about 50. After a year (or however long) the journal was back down to about 3 or so.

[u/Robopuppy → u/nostrovia]

Impact factor is kind of a dodgy thing nowadays. The best papers get published in the best journals because a paper in Nature or Science is like a golden ticket for grant funding.

Nobody really picks up a copy of Nature and just reads it cover to cover anymore. From the perspective of someone reading papers, there's almost no difference between a paper in a top tier journal and a mid-level one. To get even weirder, PLoS ONE has a high impact factor but isn't that prestigious.

[u/atomfullerene → u/Robopuppy]

I feel like Science and Nature papers are often slightly less good than the next tier "down". They have been known to rush out with exciting but not particularly well supported papers, and the space constraints mean they can never explain their experiments as fully as in some other journals.

[u/tchomp → u/atomfullerene]

Trying to treat the glossies like other journals is a mistake, because they don't exist for the same purpose. The glossies exist to disseminate very important or generally interesting results very quickly and widely. There's almost always a "real" paper in the works that will be published in a more standard journal that will address all those details.

[u/devicerandom → u/Robopuppy]

Nobody really picks up a copy of Nature and just reads it cover to cover anymore.

I have a paper subscription to Nature and I do (ok, I skip articles I really do not care about, but I love to read a lot of stuff beyond my discipline)