How Does Scientific Peer Review Work?
I recall being an energetic PhD student back in the day looking to please my thesis adviser. She wanted no less than FOUR peer reviewed papers prior to my graduation. I was not deterred, albeit I was intimidated at the prospect. Turns out I did publish one paper from my thesis work (here it is in Physiology & Behavior). Fortunately publication was not a requirement of a successful PhD thesis defense, so I did manage to graduate from Tufts! Phew.
In my experience, publication in a peer reviewed journal was a humbling process wherein a team of assigned reviewers picked apart my statistics, figures, tables, hypotheses, discussion points, word choices and overall conclusion. Humbling indeed. My adviser routinely reminded me this was a necessary part of the process to becoming a respected scientist. Bear in mind she has published hundreds of peer reviewed papers to date, so she does know a thing or two about what she speaks.
Jump ahead to present day, and I am dismayed and disheartened by the latest headlines about Dr. Brian Wansink from Cornell University, which are probing breakdowns of the peer review process:
Celebrated food researcher to step down after research questioned (CNN)
A Credibility Crisis in Food Science (The Atlantic)
Cornell Food Researcher’s Downfall Raises Large Questions for Science (NPR)
I’m not going to recap what went down (just read the articles above or many others now posted). Suffice it to say 13 retracted papers and p-hacking are major problems for the advancement of nutrition science, and all science. Yes, it appears Brian did wrong.
I had the good fortune of hearing his talks on ‘mindless eating’ during grad school. In person he was quirky, endearing and very practical. He wanted his science to make a difference for consumers. He wrote a wildly popular consumer book called Mindless Eating: Why We Eat More Than We Think to help make his findings actionable, which btw, is an admirable and often overlooked quality of published nutrition science.
How Does Scientific Peer Review Work?
Let’s break this down a bit in terms of what is supposed to happen in the land of peer reviewed science from the point-of-view of a newbie researcher like I was.
Choosing a Target Publication: Researcher picks target peer-reviewed journal to get published in (fingers crossed). These publications are sometimes called ‘refereed’ journals. Be aware not all the info in a peer-reviewed journal is subject to review. For example, editorials, letters to the editor or book reviews are not under review. Also, note that journals have an impact factor - this is a calculation designed to show the relative importance or influence of a particular journal. The higher the IF, the better. The American Journal of Clinical Nutrition routinely has among the highest impact factors of any journals in the nutrition science realm.
Submitting Initial Draft: Researcher sends off initial draft to chosen publication and it goes under review. A team of reviewers is assigned by an overseeing editor. Importantly, who is picked for serving as a reviewer is based on who is given an invitation to review. That individual can choose to accept/decline based on their expertise, interest, availability and any conflicts of interest. The process is fairly straightforward, and yet, subjectivity is definitely in the mix given the editor is choosing who to even ask in the first place. Some reviewers are tougher than others. Here’s a good flow chart of how it is supposed to work from Wiley.
Editing: After some weeks, the researcher receives the collective edits of the reviewers. Some edits may be specific text edits, specific questions on a single statistic, while others may be more directional in nature or related to the whole analysis (like - this whole study is under-powered!). This feedback incorporation step may likely include some data re-analyses. I know in my case it did. I was back in my SAS code checking and double-checking some data points based on the reviewer input. Importantly, this is the point when the much adored P value comes up. I’m not a statistician, so let me just point you to one solid attempt to explain what’s up. For our purposes, let’s focus on the fact that the goal is to be statistically significant with a p-value of <0.05. And by goal, I mean that’s the magic value at which collaborators cheer, reviewers take note and publishers happily want to publish your ‘positive’ findings.
Publishing: The draft paper is revised and resubmitted. If all goes well…it is then sent to publication by the journal, which these days usually means e-pub ahead of print. It then becomes widely available for all to freely read on sites like PubMed, though in some cases there’s a fee of approximately 30 USD to access the full paper.
It all seems simple enough. However, based on this latest situation of 13 paper retractions from a lead Cornell researcher, obviously the process is not immune to flaws and executional gaps.
Where Do We Go From Here?
My biggest wish is that we recognize the vast and overwhelming majority of peer reviewed papers are in fact, peer reviewed in thorough and proper fashion. In this rather anti-science era, this unfortunate situation with Dr. Wansink can (and will) be spun up into a bigger crisis of the credibility of all science and evidenced-based information. Let’s resist that temptation to jump into the swirl and instead see this for what it is. A clear and loud reminder that science and reviewers are imperfect. Sometimes things go awry. I suspect this controversy will ultimately lead to more rigorous and explicit measures and inspections designed to ensure peer reviewed journals, and their cadre of reviewers, can consistently and reliably produce quality science that can be trusted.