Metis Capstone project on when an MLB Manager should challenge an umpire's call

1. When should an MLB
manager challenge?
A Strategic Model
Michael Laster
Metis: September 2016
1

2. 2

3. 3

4. 4

5. 5

6. 6

7. Cespedes throws…
7

8. Right on the money!
8
but…

9. 9

10. Manager Terry Collins
challenges the call
10

11. 11
Background
• Starting in 2014, a team can “challenge” an umpire’s call
• If video reviewers find the call incorrect, it’s overturned
• Otherwise, the team loses the chance to challenge for the
rest of the game

12. Scenario II
12

13. 13

14. Now, Should Terry
Collins Challenge?
Can we make a data-driven decision?
14

15. Factors to Consider
1) How much of an advantage will a
successful challenge give them?
Without overturning: 50.0% chance of winning
If overturned: 52.2% chance of winning15

16. Factors to Consider
2) How much would it “cost” the
team if the challenge is lost for the
rest of the game?
Opportunity Cost ≈ .03% / out
remaining
In our case: 1.37%
16

17. Upside: 2.2% gain in Win Expectancy
Downside: 1.37% loss in Win Expectancy
17

18. Factors to Consider
3) Probability of Call Being Overturned
• Combined data from baseballsavant.mlb.com and retrosheets.org covering
around 3,000 replays from April 2014-July 2016
• Approximated set of non-challenged plays
•Used SVM and linear regression models to generate probabilistic estimates
• Features: challenging team, umpire, type of play, day of week, inning
• Best models: .613 and .621 log-loss
18

19. Probability
19
74.4%

20. Putting it all Together
Upside = (p)(Win Expectancy Added)
Downside = (1-p)(Opportunity Cost)
20

21. Putting it all Together
Upside = 1.6%
Downside = .35%
21

22. Challenge it!
22

23. Conclusions
•Challenge calls aggressively!
•This hinges on a proper estimate of “opportunity cost”
• The consensus is that the cost is very low
• “There's not much cost to consider.”
-Tom Tango (email correspondence)
23

24. Check out the code:
24
Github writeup

25. Thank you!
Michael Laster
michael.laster@alum.yu.edu
LinkedIN: michael-laster
25