GHME 2013 Conference Session: Dismod MR workshop Date: June 18 2013 Presenter: Hannah Peterson Institute: Institute for Health Metrics and Evaluation (IHME), University of Washington

1. Meta-regression with DisMod-MR:
how robust is the model?
June 18, 2013
Hannah M Peterson
Post-Bachelor Fellow

2. Global Burden of Disease Study 2010
2

3. YLDs
• Measures morbidity
• Requires age-specific prevalence
o For 291 outcomes
o For 2 sexes
o For 187 countries
o For 3 years
3

4. Is negative-binomial distribution
the best choice?
DisMod-MR
4

5. Alternative distributions
5
Distribution Probability Density Function
Normal
Lognormal
Binomial
Negative-
binomial

6. Alternative distributions
6
Distribution Probability Density Function
Normal
Lognormal
Binomial
Negative-
binomial

7. Alternative distributions
7
Distribution Probability Density Function
Normal
Lognormal
Binomial
Negative-
binomial

8. Alternative distributions
8
Distribution Probability Density Function
Normal
Lognormal
Binomial
Negative-
binomial

9. Potential experimental frameworks
• Data collection
o Ideal
o Impractical
• Simulation
o Impossible to know true data distribution
• Out-of-sample cross validation
o Do not have to choose distribution
9

10. Out-of-sample cross validation
10

11. Out-of-sample predictive validity
• Randomly select 25% of
data to use as “test data”
11

12. Out-of-sample predictive validity
• Randomly select 25% of
data to use as “test data”
12

13. Out-of-sample predictive validity
• Randomly select 25% of
data to use as “test data”
• Fit the remaining 75% of
data (“training data”)
13

14. Out-of-sample predictive validity
• Randomly select 25% of
data to use as “test data”
• Fit the remaining 75% of
data (“training data”)
• Use fit to calculate statistics
for test data
14

15. Out-of-sample predictive validity
• Randomly select 25% of
data to use as “test data”
• Fit the remaining 75% of
data (“training data”)
• Use fit to calculate statistics
for test data
• For each distribution
• For 1000 test-train splits
• For each disease data set
15

16. Comparing distributions
16
How to determine the best distribution?

17. Metrics of evaluation
•
17

18. Results
18
Percent of wins (%)
Distribution Bias MAE PC Total
Normal 22.1 20.6 34.6 25.7
Lognormal 29.7 13.0 36.5 26.4
Binomial 26.3 48.3 1.9 25.5
Negative-
binomial
21.9 18.1 27.1 22.4

19. Conclusions
• Choice of distribution doesn’t greatly influence results
• Best overall performance: lognormal distribution
o Contingent on method to adjust data whose value is 0
• Further investigate when each distribution performs best
o Dependent on number of covariates, priors, amount of data?
19

20. Thank you
Hannah Peterson
peterhm@uw.edu
www.healthmetricsandevaluation.org