My invited talk at the The Fifth International Workshop on Large-Scale Testing (LT 2016, co-located with ICPE).

1. Performance Regression AnalysisPerformance Regression Analysis
Cor-Paul Bezemer
Title:QueensLogo_colour.eps
Creator:Adobe Illustrator(R) 12
CreationDate:9/10/08
LanguageLevel:2
Accomplishments and ChallengesAccomplishments and Challenges

2. ““A burnt child dreads fire”A burnt child dreads fire”

3. Regression testing makes sure we doRegression testing makes sure we do
not make the same mistake twicenot make the same mistake twice
Write code
Execute
regression
test
Deploy or fix changes

4. Regression testing also helps us to makeRegression testing also helps us to make
sure we are not re-introducing more bugssure we are not re-introducing more bugs

5. Performance regression testing helpsPerformance regression testing helps
ensure we are not breaking performanceensure we are not breaking performance
Write code
Execute
performance
test
Deploy if performance is OK or fix changes

6. Performance regression testingPerformance regression testing
can/should be done on several metricscan/should be done on several metrics

7. Performance regression testingPerformance regression testing
can/should be done on several metricscan/should be done on several metrics

8. Performance regression testingPerformance regression testing
can/should be done on several metricscan/should be done on several metrics

9. Performance regression testingPerformance regression testing
can/should be done on several metricscan/should be done on several metrics

10. Directions in research onDirections in research on
performance regression analysisperformance regression analysis

11. Directions in performanceDirections in performance
regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3.Visualization

12. Execution profile comparisonExecution profile comparison

13. Execution profile comparisonExecution profile comparison

14. Disadvantages of executionDisadvantages of execution
profile comparisonprofile comparison
- Actually needs performance tests
- Need to run performance tests
several times because of variation
- Small changes may be difficult to
interpret

15. Source code analysisSource code analysis
Identify risky commits and recommend
those for testing (Huang et al. @ ICSE 2014)

16. Disadvantages of sourceDisadvantages of source
code analysiscode analysis
- Can not replace 'real' performance
measurements (yet)

17. Visualization ofVisualization of
performance regressionperformance regression
analysis resultsanalysis results

18. Control chartsControl charts
(Nguyen et al. @ MSR 2014)(Nguyen et al. @ MSR 2014)

19. Differential flame graphsDifferential flame graphs
(Bezemer et al. @ SANER 2015)(Bezemer et al. @ SANER 2015)

21. Deciding on the baselineDeciding on the baseline
performanceperformance

22. Deciding on the acceptableDeciding on the acceptable
performance deviationperformance deviation

23. Reporting results to theReporting results to the
developerdeveloper

24. Availability ofAvailability of
performance testsperformance tests

25. Execution time of testsExecution time of tests

26. Performance regression testing helpsPerformance regression testing helps
ensure we are not breaking performanceensure we are not breaking performance
Write code
Execute
performance
test
Deploy if performance is OK or fix changes

28. Directions in performanceDirections in performance
regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3.Visualization

30. Availability ofAvailability of
performance testsperformance tests

32. Execution time of testsExecution time of tests