1) Researchers developed a 4-item screening tool called the SaFETy Score to predict risk of future firearm violence using data from a cohort study of youth in Flint, Michigan. 2) The 4 predictors in the SaFETy Score assess serious fighting, number of friends carrying weapons, community gunshot exposure, and experiences being threatened with a gun. 3) When evaluated in the validation data, the SaFETy Score showed good discrimination of risk levels between scores of 0 to 10 and was a stronger predictor of future firearm violence than reason for the emergency department visit alone.

1. Development of the SaFETy Score:
A Clinical Screening Tool for Predicting Future
Firearm Violence Risk
Jason Goldstick, PhD; Patrick M. Carter MD; Maureen A. Walton MPH PhD; Linda
L. Dahlberg, PhD; Steven A. Sumner, MD, MSc; Marc A. Zimmerman PhD;
Rebecca M. Cunningham, MD

2. • Homicide is the third leading cause of death in the US among age 15-24
– 86% due to firearms
• Monetary costs of firearm violence are also substantial
– Medical and work-loss costs exceeds $2.9B per year
• Urban EDs are a critical access point for identifying high-risk youth
• Existing violence screening tools
– lack specific focus on firearm violence
– are too lengthy for practical use in a busy ED setting
Background and Motivation
Department of Emergency Medicine

3. Goal: Create a practicable basis for firearm violence prevention resource allocation
For this, we sought an empirically derived risk score that
•Can be administered quickly
– Minimal number of questions
– Easily calculated by hand
•Contains questions that can be truthfully assessed in a clinical setting
– No incriminating or embarrassing questions
We applied machine learning methods to data from a 2-year cohort study to do this
Study Objectives
Department of Emergency Medicine

4. Data came from the Flint Youth Injury study
•Patients age 14-24 arriving at Hurley ED for assault injuries were screened
•Next available non-assault-injured age- and gender-matched youth screened
•Sexual assault, child abuse, suicidal ideation were exclusion criteria
•Those reporting any past 6-month drug use were eligible
•Measurements were taken at baseline, 6, 12, 18, and 24 months
599 (349 assault injured; 250 not assault injured) enrolled
•57.3% male; 62.5% African American
•Average age = 19.9 (SD=2.4) at baseline
•Follow-up rates ranged from 83.7% to 85.3%
Flint Youth Injury study
Department of Emergency Medicine

5. Outcome: Any gun violence (victimization or perpetration) during follow-up
•483 could be ascertained, 252 (52.2%) were positive
118 candidate predictors were considered (measured at baseline)
•Received violence, including threats, and partner violence (39 items)
•Community violence exposure (5 items)
•Mental health symptoms (12 items)
•Drug and alcohol efficacy (16 items)
•Alcohol use (10 items)
•Peer and parental behaviors (21 items)
•Retaliatory attitudes and fight self efficacy (12 items)
•Age, gender, and reason for ED visit (3 items)
Measurements
Department of Emergency Medicine

6. We first split the data into a training set (75%; n=362) and validation set (25%; n=121)
On the training set, we:
1.Rank predictor importance using elastic net penalized logistic regression
– Automatically selects predictors based on prediction error using repeated cross-validation
2.Select four predictors based on important ranks and coverage of distinct content
3.Derive cut points (≤ 3 groups per variable) by optimizing AIC
4.Determine point contributions by scaling regression coefficients for the categorized predictors
Note: Multiple imputation was used at steps 1, 3, & 4 to handle the 116 missing outcomes
Risk Score Derivation
Department of Emergency Medicine

7. Variable Importance
Department of Emergency Medicine

8. The selected predictors largely fell into four domains:
•Violence victimization (peer and partner)
•Community violence exposure
•Peer/family influences
•Fighting
Predictor Domains
Department of Emergency Medicine

9. We selected one variable from each of the four domains
•Violence victimization (peer and partner)
– Frequency of being threatened with a gun
•Community violence exposure
– Frequency of hearing gunshots
•Peer/family influences
– How many of your friends carry weapons
•Fighting
– Serious fight frequency
Selecting Predictors
Department of Emergency Medicine

10. Past 6-month frequency of being threatened with a gun
•0 (“Never”), 1 (“Once”), 2+ (“Twice” or more)
Past 6-month frequency of hearing gunshots
•0-2 (“Never”, “Once or twice”, “A Few Times”), 3 (“Many times”)
How many of your friends carry weapons
•1-2 (“None” or “Some”), 3-5 (“Many”, “Most”, or “All”)
Past 6-month serious fighting frequency
•0 (Never), 1-3 (“One”, “Twice”, or “3-5 times”), 4+ (“6-10 times” or more)
Cutpoints
Department of Emergency Medicine

11. Results
SaFETy Score Calculation Rules
Department of Emergency Medicine
Response Options Score Contribution
S(Serious Fighting)
In the past 6 months, including today, how
often did you get into a serious physical
fight?
0 (never) 0
1 (once) 1
2 (twice) 1
3 (3–5 times) 1
4+ (6 or more times) 4
F(Friend Weapon Carrying)
How many of your friends have carried a
knife, razor, or gun?
1 (none) 0
2 (some) 0
3+ (many, most, or all) 1
E(Community Environment)
In the past 6 months, how often have you
heard guns being shot?
0 (never) 0
1 (once or twice) 0
2 (a few times) 0
3 (many times) 1
T(Firearm Threats)
How often, in the past 6 months, including
today, has someone pulled a gun on you?
0 (never) 0
1 (once) 3
2+ (twice or more) 4

12. • Risk gradient apparent at cut
points of 0, 1-2, 3-5, 6-8, 9-10
• Similar pattern in both training
and validation sets
SaFETy Risk Gradient
Department of Emergency Medicine

13. Generally higher scores among the
group with firearm violence
•SaFETy = 0  18.2% (2/11)
•SaFETy = 1-2  40.0% (18/45)
•SaFETy = 3-5  55.8% (24/43)
•SaFETy = 6-8  81.3% (13/16)
•SaFETy = 9-10  100.0% (6/6)
Risk Score Distribution
Department of Emergency Medicine

14. Reason for ED visit was previously
the only basis for risk assessment
•Both are individually predictive
of future firearm violence
•When SaFETy is included, ED
visit reason is no longer important
•As a standalone predictor, SaFETy
is more predictive than visit reason
SaFETy Score and Reason for ED visit
Department of Emergency Medicine
Validation Training
Model 1
Assault Injury
Model 2
SaFETy score
Model 3
Assault Injury
SaFETy score
2.14 (1.03, 4.45)
1.47 (1.23, 1.79)
1.49 (0.67, 3.32)
1.44 (1.20, 1.76)
1.89 (1.24, 2.89)
1.56 (1.41, 1.75)
1.23 (0.75, 2.00)
1.54 (1.39, 1.73)
Entries are odds ratios with 95% CIs

15. Conclusions
Department of Emergency Medicine

16. • This study is limited to a single urban ED
– Validation in other high-risk populations is required
• Our analysis was restricted to a drug-using sample
– Most (~98%) were marijuana users
• This scale is not externally validated
– Made the most of this sample but cross-validation is not magic
• Our mental health variables were not comprehensive
– Only depression and anxiety were included
– Do not want to imply that mental health is not important driver of risk
Limitations
Department of Emergency Medicine

17. • NIDA grant R01 024646 (PI: Cunningham) for funding the primary data collection
• FYI Study co-investigator team
• FYI Study staff
• Hurley Medical Center patients and staff
Acknowledgments
Department of Emergency Medicine