From:
Rob Toreki <info**At_Symbol_Here**ILPI.COM>
Subject:
Re: [DCHAS-L] Debunking Bad COVID-19 Research
Date:
Jun 30, 2020 15:07 UTC
Reply-To:
ACS Division of Chemical Health and Safety
In-Reply-To:
[DCHAS-L] Debunking Bad COVID-19 Research
Good science is reproducible. However, it's an open secret that a significant amount of biomedical (and other) research is not. Here are a couple slides from a lecture called Paradigms and Pseudoscience I do in my Nobel Prize course. Keep in mind this is discussing high-impact papers that passed peer review in some of the highest impact journals out there
Slide 1 - Good Science is Reproducible
• 2012, Nature - Amgen scientists look at 53 landmark cancer studies. Confirmed only 6 (11%).
• 2011 - Bayer Scientists found only 25% of published preclinical studies could be validated.
• These papers spawned hundreds of other secondary studies that did not seek to confirm or falsify the original work.
• Secondary research included clinical studies. Wow.
• Reproducible studies- "authors had paid close attention to controls, reagents, investigator bias and describing the complete data set."
• Others - plagued by lack of double blind control studies, presentation of a single result or data point, supplying data that supports their hypothesis but discarding data that does not!
Slide 2 - This is Widespread
• NIH official comments 75% of published biomedical findings would be hard to reproduce.
• Smaller studies more prone to false conclusions.
• Nobody gets a publication/credit for reproducing work.
• "Publish or perish" makes people push out work prematurely.
• Reviewers seldom look at supplemental material and (in chemistry) do not reproduce the experiments.
See
"Unreliable Research: Trouble At The Lab", The Economist, 2013, Oct 19th 2013
http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
Slide 3 - The Math (using figures from the above-see graphic titled "Unlikely results")
• Assume 1,000 hypotheses, of which 100 are true.
• Assume false negatives will result in 80% (80) of them being found.
• Of 900 false hypotheses, 5% (45) will be false positives for various reasons.
• This makes 125 positive results, so (45/125) = 36% of the results are bogus!
• Negative results are 97% trustworthy.
• But no journals are interested in negative results!
Key poins - you don't get tenure, you don't get grants, and your company doesn't make money for replicating someone else's research. And this leads to a dangerous cascade of research and effort based on unconfirmed results.
Ask me about peer review next-.
Rob Toreki
======================================================
Safety Emporium - Lab & Safety Supplies featuring brand names
Fax: (856) 553-6154, PO Box 1003, Blackwood, NJ 08012
It will be interesting to see if this approach of peer review after publication becomes a trend in the scientific publishing world as in person meetings become less common and preprints rise in prominence in all fields...
