COVID-19 Day 65: The Pitfalls of Scientific Peer Review

We rely on scientists in these uncertain times for the ‘truth’. But as Indiana Jones said, if you want the truth, ask a philosopher, not a scientist. 

As a guy from the corporate world, I never knew the academic world was so cut-throat or that peer review had a dark underbelly. I thought a peer-reviewed scientific paper meant that it had passed some 5-point quality inspection for accuracy and trustworthiness. But as we’re discovering with the rapid revisions in the understanding of COVID-19, the science often isn’t settled. And that peer-review isn’t a guarantee of excellence, it’s just a fence to keep the riff-raff out of the pool.

In a paper, “Peer review: a flawed process at the heart of science and journals” published in the Journal of the Royal Society of Medicine, the author (a former editor of the journal) bemoans its many shortcomings. The classic peer-review process itself is not a guarantee of the quality of the papers published. And perhaps a broader crowd-sourced method similar to Amazon reviews or Reddit could be employed?

Scientific journals publish research papers submitted to them by scientists. But before a paper accepts and publishes that paper, it first sends out copies to be reviewed by other experts in that field, ‘Peers’. If the reviewer finds faults in the method, content, or just doesn’t think the information is significant, the journal will often reject the paper.

A big flaw in the process is that most reviews are unpaid. So scientists must volunteer their time to review papers, out of their own sense of duty (or mutual benefit). Reviewing papers often fall to the back-burner for scientists; behind teaching or their own research. As a consequence, it can take months, even years for a submitted paper to be published or even rejected.

Peer-review is also subject to personal politics, conflicts, and rivalries. Scientists rely on grants to fund their research. Grants are more often awarded to the scientists who are perceived to be the more successful and noteworthy because they have published the most papers. ‘Publish or perish’ is the academic axiom.

The reviewers are supposed to be anonymous but scientific fields are small worlds. Scientists tend to know what kind of work their peers are doing, so inferring the identity of a peer-reviewer is apparently not difficult. Scientists compete with each other for grant money and have a vested interest in promoting the papers of their friends and colleagues over the papers of rivals.

This is not a new phenomenon. Feuds involving history’s most famous scientists like Isaac Newton, Nikola Tesla, and Jonas Salk, reveal personal enmities and backstabbing snipes that would make for good Soap Opera. When a junior scientist submits a paper for peer review they also run the risk of having rival scientists reject the work to stifle them or outright steal it.

On a podcast with the evolutionary biologist, Brett Weinstein, he revealed for the first time, a story of scholarly theft that happened to him decades ago and sidelined his career. He co-authored a paper and submitted it to the prestigious journal, Nature and was rejected. He discovered his submission was blocked by a famous molecular biologist, with whom Brett had previously corresponded to help with his research. What was more galling was that she would later use Brett’s findings in her own lectures, after she was awarded a Nobel prize for science.

The peer-review process has even more consequences than a scientist’s career. Government agencies rely on peer-reviewed papers in promoting drugs like Remdesivir or Hydroxychloroquine. We’re also seeing a politicization of science with the shut-down vs. re-open camps debating models of herd immunity and COVID-19 death numbers. And we can’t forget climate science.

The concern is that non-scientists (like myself) make assumptions as to the validity of scientific papers we come across. Especially if those non-scientists are journalists who often turn ‘implications’ into ‘absolutes’ for sensational, click-bait headlines. I’m old enough to remember the media hype around  ‘Cold Fusion’. This is why scientists and journal editors are focused on trying to get things right before a paper is published. A retraction is far more damaging to the scientific confidence than a rejection.

With all of its faults and biases, peer-review is only one step in the process of disseminating science. Once reviewed by the journal’s editors, who are also scientists, a peer-reviewed paper may be published. The scientific community that reads that journal, traditionally voice their own opinions out in the open. Some conduct confirmation research to disprove or build upon the original papers’ findings, and in turn submit their own papers. The process goes on.

Science is not truth. It’s an iterative process of testing and skepticism that attempts to discover a greater sense of certainty. But like all human attempts, it reflects the humans that engage in it; all of our passions, pettiness, and prejudices. We should all aspire to greater objectivity in science. But the fault lies not in our stars but in ourselves.


Comments are closed.