Upload
national-academies-of-science-engineering-and-medicine
View
221
Download
0
Embed Size (px)
Citation preview
8/12/2019 Glenn Begley ILAR Presentations
1/16
Great Expectations A Critical Assessment ofPublished Research:
can you believe everything you read?
C. Glenn Begley MB BS, PhD, FRACP, FRCPA, FRCPath
CSO & SVP TetraLogic PharmaceuticalsJune 4, [email protected]
8/12/2019 Glenn Begley ILAR Presentations
2/16
Disclosure InformationAACR Annual Meeting 2014
C. Glenn Begley,
CSO TetraLogic Pharmaceuticals, Malvern, PA- employee and stock holder
Non-Executive Director at Oxford BioTherapeutics, UKSAB member for non-public biotech companies and academic institutions
2002-2012: VP Hematology and Oncology Research, Amgen- continue to hold Amgen stock
Prior to Amgen: >20 years; physician/scientist (hemato-oncologist),department head
I will not discuss off-label use and/or investigational use of drugs
8/12/2019 Glenn Begley ILAR Presentations
3/16
Begleys position statement
These results do not challenge the validity orlegitimacy of the scientific method
Not talking about fraud: the subject isscientific-laziness, sloppiness, ignorance,exaggeration, desperation
The vast majority of investigators want to do
the right thing That this debate is occurring in public confirms
the strength our scientific system
8/12/2019 Glenn Begley ILAR Presentations
4/16
Cancer is Evolution
8/12/2019 Glenn Begley ILAR Presentations
5/16
8/12/2019 Glenn Begley ILAR Presentations
6/16
The opportunity for cancer research1) Challenge: Inherent to the disease, and extremely difficult to address
a. Cell lines are artificial
b. Animal models are artificialc. Human cancers are heterogeneous in practically every dimension we care to think about:
i. Tissue of originii. Genetic alterationsiii. Host adaptations and interactionsiv. Genetic instabilityv. Pathways of escape from growth control
vi. Invasiveness and metastasis potential. etc
2) Opportunity: Inherent to our system, and more easily addresseda. Poor experimental design i. Lack of blinding
ii. Lack of adequate controlsiii. Lack of prospective hypothesis statementiv. Lack of appropriate statistical power
b. Poor reagentsc. Poor analysis e.g. Inappropriate statistical methodsd. Failure to reject hypothesis after observing discordant, valid experimental resultse. Deliberate bias in selecting positive rather than negative results to report, publish, cite, fund
i. Scientistsii. Journal editorsiii. Funding agencies
iv. Pressf. Failure to reject our favorite hypothesis
8/12/2019 Glenn Begley ILAR Presentations
7/16
An unappreciated challenge tooncology drug discovery:
publication bias
Sometimes we can set the bar But we get what we incentivize
8/12/2019 Glenn Begley ILAR Presentations
8/16
Industry relies heavily upon
targets and pathways identified byacademic groups
Between 2002-2012, Amgen was not able toreproduce 47of 53 seminal publications.
These were publications that reportedsomething completely new
(not binary publications)
8/12/2019 Glenn Begley ILAR Presentations
9/16
The spectrum of irreproducibility
data could not reproduced by the originalinvestigators with their reagents in their lab
specific data reproduced, but not a generalfinding
data selection bias: a single, non-
representative experiment was reported
Investigators often required Amgen to sign a Confidentiality Agreement to allow
an exchange of reagents and to allow Amgen scientists to work in their labs
8/12/2019 Glenn Begley ILAR Presentations
10/16
These studies have had substantial impact
Wasted effort: multiple investigators,multiple companies,opportunity cost
Some papers have spawned a whole field with hundreds ofsecondary publications
Clinical studies initiated
8/12/2019 Glenn Begley ILAR Presentations
11/16
only in approx. 20 25% of the projects were therelevant published data completely in line with our in-
house findings..in most cases (this) resulted in
termination of the projects because the evidence .wasinsufficient to justify further investments into these
projects
Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data
on potential drug targets? Nat Rev Drug Discov 10, 712,
Amgens experience is, unfortunately, not unique
8/12/2019 Glenn Begley ILAR Presentations
12/16
Preclinical irreproducibility is a systemic problem,driven by current incentives
Careers are built on publications in top -tier journals: drive grants, fame, promotion
Top- tier journals want the best story simple, clear, compelling
Positive studies are rewarded:little recognition of value of negative studies,
reward for finding the answer a Reviewer wants/expects
8/12/2019 Glenn Begley ILAR Presentations
13/16
The Unspoken, Unacknowledged Disclosure Information:the real, inescapable f inancial driver
and
major driver of irreproducibility
C. Glenn Begley,CSO TetraLogic Pharmaceuticals, Malvern, PA
Begley CG & Ellis LMDrug development: Raise standards for preclinical cancer research
Nature 483, 531-533 (2012)Begley CG
Six Red Flags for Suspect Work
Nature 497, 433-434, (2013) The test of time and editorial responsibility .
Pigment Cell and Mel. Res . 26: 603 604 (2013)
Presidents Council of Advisors on Science and Technology
8/12/2019 Glenn Begley ILAR Presentations
14/16
raising the standards of Grants & Publications
encouraging publication of confirmatory data
rewarding findings that refute high-profile studies
The Investigator and host institution are ultimatelyresponsible and accountable,
but the greatest likelihood for changewill come from granting agencies and journals
8/12/2019 Glenn Begley ILAR Presentations
15/16
High-profile studies typically fail at multiple levels:
1) Were studies blinded?Almost never
2) Were all results shown?Typically not representative examples & data selection bias
western blots that show only a slice; no size markers
3) Were experiments repeated?Often not westerns/immuno-precipitation usually only performed once
typically only use 1/2 siRNAs and in 1/2 cell linesconfusion between replicates and independent experiments
4) Were positive and negative controls shown?Typically not
5) Were reagents validated?Frequently not IHC with a polyclonal anti-peptide Ab
small molecule inhibitors
6) Were the statistical tests appropriate?Typically not
Begleys six criteria for judging scientific reports:
Nature 497, 433-434, 23 May 2013
8/12/2019 Glenn Begley ILAR Presentations
16/16
Reviewers, editors of top -tier journals
repeatedly tolerate poor quality science
These studies typically fail at multiple levels
Three random examplesbut you can find more every time you open a top -tier journal