Researcher KnowHow session presented by Carrol Gamble, Anna Kearney and Paula Williamson, Department of Health Data Science. University of Liverpool and Trials Methodology Research Partnership.
3. Research Waste/Adding Value
Lancet 2009; 374:86-89
“implies that the dividends from tens of billions of dollars of investment in research are lost every year because of correctable problems”
4. “When I take a look at the data I see what best
advances the story, and if you include too much
data the reader doesn’t get the actual important
message, so sometimes you get data that is either
not significant or doesn’t show anything, and so
you, we, just didn’t include that”
Smyth et al. BMJ 2011; 342:c7153.
5. Why are outcomes important?
• Interventions are compared in Randomised Control
Trials (RCTs) by measuring differences in patient
outcomes between the groups
• ‘Clinical trials are only as credible as their outcomes’
(Tugwell 1993)
6.
7. • Fully reported outcome data
- odds ratio 2.2 to 4.7 if statistically significant results
• Reports compared to protocols
- 40–62% at least one primary outcome changed, newly
introduced or omitted
8. BMJ (2010); 340:c356
• Outcome reporting bias (ORB) suspected in at least one trial
in 34% of 283 Cochrane reviews
9. • 42 significant meta-analyses
8 (19%) would not have remained significant
11 (26%) would have overestimated the treatment effect by > 20%
10. • Poor reporting indicates a collective failure of authors, peer
reviewers, and editors
• Researchers (authors) may not know what information to
include in a report of research
• Editors may not know what information should be included
What help can be given to authors?
What help can be given to editors?
Whose fault is poor reporting?
11. • Prospective trial registration
• Protocol required
• Transparency declaration
• 20% pre-specified outcomes N/R
• 10% new outcomes introduced
• Reasons:
• Space limitation
• Outcomes not yet analysed
• Reported elsewhere
• Errors
Articles submitted
September 2013 – July
2014
(n=3156)
Non-RCT articles
excluded (n=2845)
RCTs identified
(n=311)
RCTs excluded (n=36)
Follow-up studies: 6
Re-analysis of RCT data:
10 Cost-effectiveness: 12
Secondary RCT paper: 5
Critique of RCT: 1
Diagnostic test accuracy: 2
RCTs included
(n=275)
Accepted by BMJ (n=21)
Protocols
Yes (n=21)
No (n=0)
Rejected by BMJ (n=254)
Protocols
Yes (n=115)
No (n=139)
12. • Statistical Analyses: 7% to 88%
• Composite outcomes: 33%
• Subgroup analyses: 61% to 100%
• Dichotomising continuous outcomes: 1% to 29%
• Dealing with missing data: 12% to 80%
• Unadjusted versus adjusted analyses: 46% to 82%
13. N=99 Adequately Incompletely Not mentioned
Analysis Population 8 (8%) 64 (65%) 27 (27%)
Analysis Model 27 (27%) 61 (62%) 11 (11%)
Covariates 40 (40%) 32 (32%) 27 (27%)
Handling of missing data 10 (10%) 24 (24%) 65 (66%)
No. of aspects
adequately
defined
No. (%)
0 39 (39%)
1 36 (36%)
2 23 (23%)
3 1 (1%)
4 0 (0%)
14. • 101 eligible trials in 6 leading general medical journals
• 88% had a publicly available pre-specified analysis
plan
– Only 25% had no unexplained discrepancies
– 61% had ≥ 1 unexplained discrepancy
– 15% it was impossible to determine
• Conclusion: Unexplained discrepancies are common.
Increased transparency is required
15. Protocol pre-specification of the
analysis plan
If a Statistical Analysis
Plan is to be produced
separately, state this here
and condense the most
relevant information from
the sub sections here.
(NHS HRA protocol
template)
16. What is a Statistical Analysis Plan (SAP)?
“a document that contains a more technical and detailed
elaboration of the principal features stated in the protocol
and includes detailed procedures for executing the
statistical analysis of the primary and secondary variables
and other data”. (ICH E9)
• Similar to protocols, the ability of a SAP to provide
transparency is dependent on its content
17. Why bother with a SAP?
• Legal requirement to comply with trial protocol
• SAP not required when the protocol contains all
necessary information
• Aim is replication by an independent statistician- no
ambiguity
– SAP is not just analysis but also data
manipulations/calculations/derivations
• Protocol audience appropriate
• Could lead to unnecessary protocol amendments
18. - Special communication in JAMA in
2017 supported by an editorial
JAMA. 2017;318(23):2337-2343.
doi:10.1001/jama.2017.18556
SAP guidance
19. • Assumptions:
– The SAP is not a standalone document and should be read in
conjunction with the clinical trial protocol
– The protocol should be consistent with SPIRIT
– The SAP is to be applied to a clean or validated data set for analysis.
• 6 sections covering 32 items
• Elaboration document available (JAMA)
• Checklist available (https://lctc.org.uk/SAP-Statement)
20. In advance – but when?
• SAP may be written after finalising the protocol
– Open study: before first participant is recruited
– Blinded study: before interim analysis
• Version controlled with revision history
• SAP should be followed & changes justified &
reported
• Example: Lancet 2019; 393: 2125–34
– centre could not be included in the Cox model due to lack of
convergence.
21. SAPs do not prevent post-hoc analyses
• Analyses performed in the light of the data that were
collected (rather than being of interest before data
collection began)
• Sometimes requested – by research team, by journal
• Transparency is key – declare post-hoc analyses as
such, together with rationale
• Example of statement: Lancet 2019; 393: 2125–34
A post-hoc analysis was undertaken for the reasons underlying the
further management of the presenting episode of convulsive status
epilepticus, the assessment of which was done without knowledge of
the allocated intervention.
22. Extensions
• Developed for later phase randomised trials
• Early phase extension in progress
• Extensions to:
– Adaptive designs
– Bayesian
• Observational studies
25. Impact
• Actively used within UK Clinical Trial Units
• Documents accessed from EQUATOR website and
www.lctc.org.uk/SAP-Statement (Dec 2019-Oct 2020)
– 141 downloads of the checklist
– 77 downloads of the elaboration document
• JAMA article viewed 80k times with 22,347 pdf
downloads (13th Nov 2020)
26. 124 citations identified (71 from JAMA via Web of Science /121 from Google scholar identified on 10th November 2020)
Citation Analysis
* 15 citations excluded (7 duplicates, 4 awaiting access, 1 citation could not be confirmed and 3 editorial /letters associated with the original publication)
Primary reason for citation 2018 2019 2020 Grand
Total
Guidance used as a template 6 8 28 42
Reporting and transparency 7 8 11 26
Recommending use of guidance 1 9 10
Future SAP will follow guidelines 1 4 3 8
Discussion of statistical methods 2 2 2 6
Justification for a SAP element 1 3 2 6
Other guideline development 1 1 2 4
Other 1 2 3
Application outside RCT 2 2
Data sharing methods 1 1
Cross referencing guidelines 1 1
Grand Total 19 30 60 109*
Country of first
author
UK 15
Denmark 6
Australia 4
USA 4
Canada 3
Netherlands 2
unknown 2
Saudi Arabia 1
China 1
Chile 1
Italy 1
Finland 1
Germany 1
27. ICMJE guidance (2014): “As part of peer review,
editors are encouraged to review research
protocols, plans for statistical analysis if separate
from the protocol, …… encourage authors to
make such documents publicly available at the
time of or after publication, …. ”
Impact: Journals
28. Impact: Journals
• RCT policies of major medical journal align ICMJE
• Only the BMJ referenced the SAP guidance
• Unclear how SAPs are used
• SAPs for RCTs published in 2016 (Spence 2020)
- <10% for AIM, BMJ, Lancet
- 53% JAMA
- 92% NEJM
30. Impact: Funders
• Guidance soon to be added to:
– Wellcome Trust RCT funding policy
– NIHR NETSCC data sharing policy
• Guidance highlighted in 2020 online EViR funders
conference
31. • Failure to translate – differences in underlying
biology, bias in design/conduct/analysis/reporting
• 4445 comparisons: 919 expected, 1719 observed
• “selective analysis and outcome reporting biases
being plausible explanations”
• Preregistration, access to protocol/data/analysis,
ARRIVE reporting guidelines
32. • “unacceptably easy to publish statistically significant
evidence consistent with any hypothesis”
• Culprit: “researcher degrees of freedom”
• Chronological rejuvenation experiment (“people were
nearly a year-and-a-half younger after listening to When
I’m Sixty-Four”)
• Recommend a disclosure-based solution
33. Prevalence of systematic reviews that added, omitted,
upgraded or downgraded at least one outcome between the
protocol and published systematic review was 38% (95% CI
23% to 54%)
34. Discussion
• “Our goal as scientists is not to publish as many
articles as we can, but to discover and disseminate
truth” (Simmons et al)
• “We need less research, better research, and research
done for the right reasons” (Altman)
• Therapeutic statistical solutions exist but prophylactic
ones are better
• Statistical Analysis Plans
• Transparency is key
