Tom Pollard talks about reproducibility in critical care research & makes an introduction to MIMIC, the eICU Collaborative Research Database and datathons Workshop title: Datathons in Evidence-Based Medicine: Applying Open Science Principles to Support Cross-Disciplinary Education and Research Workshop abstract: In this interactive workshop, we explore how open science enables “datathons”, events that bring together teams of researchers to work together on unanswered clinical questions. We begin by outlining the datathon model and describe our experiences in holding these events internationally. We then offer an opportunity to participate in an interactice exercise, working together to analyse highly detailed information collected from patients admitted to critical care units at a large tertiary care hospital. Participants will learn about open science in clinical research and gain an overview of MIMIC-III, a freely-available critical care dataset collected from over >50,000 hospital stays. DAY 2 - PARALLEL SESSION 3 http://opensciencefair.eu/training/parallel-day-2-1/datathons-in-evidence-based-medicine-applying-open-science-principles-to-support-cross-disciplinary-education-and-research

1. Reproducibility in
critical care research
Dr Tom Pollard
Laboratory for Computational Physiology,
Massachusetts Institute of Technology (MIT), USA
@tompollard

2. Critical
care

3. Heart rate
O2 saturation
NIBP, mean
Respiratory rate
Intake volume, dL
Output volume, dL
0
0
10 30 504020 60
120
100
80
60
40
20
Time after admission to the intensive care unit, hours
Measurement,absolutevalue
Code status Full code Comfort measures
Incomprehensiblesounds
Flex-withdraws
None
Oriented
Obeys commands
Spontaneously
Oriented
Obeyscommands
Spontaneously
Oriented
Obeys commands
Spontaneously
Confused
Obeyscommands
To speech
Confused
Obeyscommands
To speech
GCS: Verbal
GCS: Moto
GCS: Eye
Platelet, K/uL
Creatinine, mg/dL
Whitebloodcell,K/uL
Neutrophil, %
Morphine Sulfate
Vancomycin(1dose)
Piperacillin(1dose)
NaCl 0.9%
Amiodarone
Dextrose 5%
48
0.7
9.1
37
53
12.4
46
0.7
16.8
45
0.8
23.2
10.0mL/hour 10.0mL/hour 10.0mL/hour
1mg/min 0.5mg/min 0.5mg/min
50mL/hour 25mL/hour 25mL/hour

4. 4
Title
4

5. Research opportunity
■ …data that could be used to discover
new knowledge for the benefit of patients
5

6. 6
■ but, this data is inaccessible to
researchers

8. 8
http://mimic.physionet.org

9. 9
Two key steps to gaining access:
• complete a recognized course in protecting
human research participants that covers Health
Insurance Portability and Accountability Act
(HIPAA) requirements
• sign a data use agreement, which outlines
appropriate data usage and security standards,
and forbids efforts to identify individual patients.
Accessing the data
9

11. eICU Collaborative Research Database

12. https://eicu-crd.mit.edu

13. Relational database
(a collection of linked spreadsheets)
1313

15. 28 courses worldwide,
and counting…

17. Reproducibility is the ability to reproduce the
results of a given study
Note the distinction between this and
*replication* (whether the results hold up in
different experimental conditions).
Reproducibility

18. Nature 533, 452–454 (26 May 2016) doi:
10.1038/533452a

19. Why should I care about
reproducibility?

21. Reproducibility leads to incremental progress

22. Incremental progress on ImageNet
A. Canziani et al, “An Analysis of Deep Neural Network Models for Practical
Applications”, CoRR, 2016.

23. MIMIC is freely available
critical care database
So papers using the MIMIC
are reproducible. Right?

24. Reproducibility in critical care:
a mortality prediction case study
● Collect all studies which attempted to predict mortality in
recent history
● Attempt to regenerate the cohort
● Compare our reproduced study cohort with the published
cohort
Reproducibility in critical care: a mortality prediction case study.
Proceedings of Machine Learning for Healthcare (2017).

25. e.g. presence of
bilateral
infiltrates on
chest
radiograph
38 distinct evals.

26. Methods
1. Define a base cohort
2. For each study, add in the additional exclusion criteria
specified
3. Compare the published sample size and mortality rate to
ours
4. **Bonus** compare simple logistic regression AUROC to
their AUROC

27. Base cohort
● Start with a “base” cohort with the minimum required exclusions
● Exclude
○ Patients < 15 years old
○ Invalid admissions (no charted obs, no heart rate obs, no
admit/disch time)
○ Organ donor accounts
○ Stays less than 4 hours
● Outcomes of interest
○ In-hospital mortality
○ Post ICU discharge mortality
■ 48-hour, 30-day
○ Post hospital discharge mortality
■ 30-day, 6-month, 1-year, 2-year

28. Additional cohort criteria
● Hug et al. 2009
○ >1 obs. for HR/GCS/Hct/BUN, Not NSICU/CSICU, first
ICU stay, full code, no eventual brain death
● Lee et al. 2015, Lee and Maslove 2017
○ Only ICU stays with complete SAPS data
● Ghassemi et al. 2014
○ Age>18, >100 words across all notes
● Grnarova et al. 2016
○ Age > 18, stays with only one hospital admission
● Che et al. 2016
○ None described

29. Results - sample size

30. Results - sample size
75%
have
1000+
more
patien
ts

31. Results - % mortality

32. Results - AUROC
71% of logistic
regression models
outperformed
published paper

33. Discussion
● The majority of studies were not
reproducible (>2/3rds)!
● How can we make it easier for others to
reproduce our papers?

34. Discussion
● State all restrictions to cohort
○ Age, length of stay, certain care units, certain diagnoses
● Be explicit in criteria description
○ Specify MICU service or MICU physical location
○ “Removed patients with missing data” -> BAD!!!
○ “Removed patients with fewer than 1 heart rate
measurement” -> GOOD!!!
● Detail data abstraction steps
○ 64% of studies were outperformed by logistic regression
on simple features
○ Clearly data abstraction matters - give it more space in the
paper!

35. Reproducibility
Reusable
data

36. Progress on ImageNet not just thanks to data…
A. Canziani, A. Paszke and E. Culurciello, “An Analysis of Deep Neural Network Models for
Practical Applications”, CoRR, 2016.

38. Reproducibility
● Reproducibility has two components
Reusable
data Reusable
code
Please do this more!

39. Thank you for your attention!