About This Archive | DCHAS-L 2020 Index | DCHAS-L Yearly Index | DCHAS-L Home Page
From: Reba Surma-Heine <rebasurma**At_Symbol_Here**GMAIL.COM>
Subject: Re: [DCHAS-L] Debunking Bad COVID-19 Research
Date: Jul 1, 2020 20:08 UTC
Reply-To:
ACS Division of Chemical Health and Safety
In-Reply-To: Re: [DCHAS-L] Debunking Bad COVID-19 Research
Good science is reproducible. However, it's an open secret that a significant amount of biomedical (and other) research is not. Here are a couple slides from a lecture called Paradigms and Pseudoscience I do in my Nobel Prize course. Keep in mind this is discussing high-impact papers that passed peer review in some of the highest impact journals out there
Slide 1 - Good Science is Reproducible
• 2012, Nature - Amgen scientists look at 53 landmark cancer studies. Confirmed only 6 (11%).
• 2011 - Bayer Scientists found only 25% of published preclinical studies could be validated.
• These papers spawned hundreds of other secondary studies that did not seek to confirm or falsify the original work.
• Secondary research included clinical studies. Wow.
• Reproducible studies- "authors had paid close attention to controls, reagents, investigator bias and describing the complete data set."
• Others - plagued by lack of double blind control studies, presentation of a single result or data point, supplying data that supports their hypothesis but discarding data that does not!
(see a summary of this type of stuff here: https://www.theatlantic.com/science/archive/2017/01/what-proportion-of-cancer-studies-are-reliable/513485/ )
Slide 2 - This is Widespread
• NIH official comments 75% of published biomedical findings would be hard to reproduce.
• Smaller studies more prone to false conclusions.
• Nobody gets a publication/credit for reproducing work.
• "Publish or perish" makes people push out work prematurely.
• Reviewers seldom look at supplemental material and (in chemistry) do not reproduce the experiments.
See"Unreliable Research: Trouble At The Lab", The Economist, 2013, Oct 19th 2013
http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-troubleSlide 3 - The Math (using figures from the above…see graphic titled "Unlikely results")
• Assume 1,000 hypotheses, of which 100 are true.
• Assume false negatives will result in 80% (80) of them being found.
• Of 900 false hypotheses, 5% (45) will be false positives for various reasons.
• This makes 125 positive results, so (45/125) = 36% of the results are bogus!
• Negative results are 97% trustworthy.
• But no journals are interested in negative results!
Key poins - you don't get tenure, you don't get grants, and your company doesn't make money for replicating someone else's research. And this leads to a dangerous cascade of research and effort based on unconfirmed results.
Ask me about peer review next….
Rob Toreki
======================================================Safety Emporium - Lab & Safety Supplies featuring brand namesyou know and trust. Visit us at http://www.SafetyEmporium.comesales**At_Symbol_Here**safetyemporium.com or toll-free: (866) 326-5412Fax: (856) 553-6154, PO Box 1003, Blackwood, NJ 08012
On Jun 30, 2020, at 10:46 AM, Ralph Stuart <ralph**At_Symbol_Here**rstuartcih.org> wrote:
It will be interesting to see if this approach of peer review after publication becomes a trend in the scientific publishing world as in person meetings become less common and preprints rise in prominence in all fields...
- Ralph
Begin forwarded message:
From: Robert C Michaelson <rmichael**At_Symbol_Here**northwestern.edu>
Subject: [CHMINF-L] Debunking Bad COVID-19 Research
Date: June 29, 2020 at 4:15:33 PM GMT-4
From today's Inside Higher Ed